3 ms·

Can't claim I read the entire thing, but I got down to methodology and must say it doesn't look particularly impressive. The study is a survey sent out strictly

by Kranar 6y ago

Can't claim I read the entire thing, but I got down to methodology and must say it doesn't look particularly impressive. The study is a survey sent out strictly to Microsoft employees asking them how they rank a set of pre-defined criteria about what makes a great software engineer. That criteria includes things like "hard working", "honesty", "team player", "creates a safe haven" etc... Obviously no Microsoft employee is going to say something like "Lazy liars who treat people like crap make great software engineers." At the very least you'd expect a study to have a control group in order to filter out these useless questions. For example send a similarly worded survey to a group of grocery cashiers and ask them if they think attributes such as "Hard work, honesty, integrity, long term thinking, being a team player." make someone a great engineer. My hypothesis is you'd end up getting similar answers from taxi drivers as you would from Microsoft employees and consequently that there isn't much value derived from this survey or their selection of participants. Anyways, I'm by no means an expert and could be misjudging this research but frankly it looks to be beyond useless.

glushkov 6y ago
Most academic software engineering “research” I’ve seen is pretty useless, especially these kinds of human factors studies. Questionnaires and interviews are usually too underpowered or specific to be statistically significant, and their terms are defined vaguely enough to draw any conclusions the authors want. Occasionally there is some interesting demographic information, but most of the papers I’ve read are usually glorified position papers under the guise of an empirical study.
- kqr 6y ago
  What is your take on the DevOps DORA research? From what I could tell of the explanations of methodology in Accelerate (as well as Jez Humble and/or Nicole Forsgren talking about it) they seem to do relatively solid survey-based work.
brentsch 6y ago
The paper may not offer insightful prescriptions for experienced engineers, but can work like this still be useful for informing future studies in a meaningful way? The authors repeatedly note the widespread inadequacies of the current research landscape. (To anyone familiar with the literature, is their assessment accurate?) In my eyes, the message is that the paper represents an incremental step in the direction of a truly detailed understanding of the factors involved in developer productivity. Even if there's a broad intersection between the answers one would get from taxi drivers and from computer scientists, the distance between the two fields makes that an unexpected result, which should prompt us to change how we think about computer science* and/or how we think about studying it. *or cab driving
- bdowling 6y ago
  > The authors repeatedly note the widespread inadequacies of the current research landscape. This is standard language in academic research papers. It is there to sell the importance of the research to the reader, in particular to journal editors or peers who review the article. It is mere puffery.
  - deleted 6y ago
    [deleted]
texasbigdata 6y ago
In grad school we had a forced module on information systems in HR in which one case was about RBC and their new formerly-Bain Director of HR Data and the 3 or 4 year effort to “quantify” labor. Cases should be taken with some skepticism but let’s assume the case was correct. I wanted to tear my hair out. Not only was the output or ultimate deliverable of this gargantuan activity an infographic of largely non falsifiable random words (trustworthy, reliable, predictable, etc etc), I did a quick back of the envelope that the approach to repeatedly solicit feedback on every employee from every business line and every customer they intersect could generate 5 trillion outbound survey “friction” onto their customers and stakeholders in a given year give or take 2 orders of magnitude. What does that cost in terms of non-enjoyment? Does it even work, and at what response rate at that scale? And what does it cost to maintain, since surely the Canadian treasury at least sort of requires or at worst encourages banks to employ trustworthy employees. Worse, how implementable are the outcome and/insights? For a real open requisition is evaluating “trustworthiness” at time of hire possible? Is it possible to compare two candidates on a relative or absolute basis on this metric? And if they are hired, going forward how is that tracked? More surveys? Perhaps the case wasn’t as bad, and I’m sure the OP HB post of this paper has subtleties in survey design or methods I don’t appreciate, but just to pick on them by cherry picking the first few attributes that stick out: passionate, systemic, data driven, focused, hardworking, persevering, etc?!?!? Worse, where’s the actual data? Like when Google apparently decided to get rid of middle management it felt like a reversible scientific experiment with measurable outcomes and metrics. This doesn’t. For example, from personal experience, in investment banking being hard working especially at the very lowest and newest ranks is similarly lauded and required and emphasized. But with a fairly simple app that tracks second level activity of which app is open and when and can assign a rule a rule based score, when reviewing a few years of data, the data contradicted the folsky saying. The data showed above XX hours per week for me was non-restorative, and further above XX hours would create some incremental inefficiency increase, and above even that was a “tech debt” like factor which later required recuperation (which could also be calculated but maybe that was an accident). In English: if (simple numbers) working 10 hours leads to 8 hours of work and 2 hours of goof off for some productivity, maybe working 14 hours is 10/4, and maybe working 17 is 11/6 and also requires 2 hours of rest at a subsequent trailing period. Depending on what’s going on, maybe it makes sense to do that, maybe it doesn’t. The software engineer view would be much more interesting. It would be fascinating to compare even arbitrary metrics of consenting individuals across a large pool. Who gets the biggest bonus relative to hours worked? What time do peak performers come in and leave at and are there any wfh days? Even trivial but theoretically transferable insights would help: “what percent of top performers drink no caffeine or tea” or “pair programming and velocity”. Heck, buzzfeed baiting analysis like “do top engineers type faster? A 10 part slideshow on why you should switch to a mechanical Dvorak keyboard” or “Can you type XYZ words per minute? Find out if you are dragging your team down”. Anything but more lengthy repetitions of “trust” and “respect” and whatever the word of the day is.
sokoloff 6y ago
One of the key takeaways from the paper for me is the relative ranking of those 54 attributes. If the employees ranked “long-term thinking” above “creates a safe haven”, I have possibly learned something. If engineers rank attributes differently than managers or if cashiers rank attributes differently than tech workers... Side note: I also thought the negative phrasing methodology used was interesting. They asked “is it possible to be a great engineer without this quality?” which I think is most tuned towards qualifying and ranking attributes. Will it tell you individually how to become a great engineer? No, there are hundreds of posts, many on medium.com for that. This paper may not be perfect, but I found a lot to consider inside and “beyond useless” is pretty far from my judgment.

by Kranar 6y ago

glushkov 6y ago

Most academic software engineering “research” I’ve seen is pretty useless, especially these kinds of human factors studies. Questionnaires and interviews are usually too underpowered or specific to be statistically significant, and their terms are defined vaguely enough to draw any conclusions the authors want. Occasionally there is some interesting demographic information, but most of the papers I’ve read are usually glorified position papers under the guise of an empirical study.

kqr 6y ago
What is your take on the DevOps DORA research? From what I could tell of the explanations of methodology in Accelerate (as well as Jez Humble and/or Nicole Forsgren talking about it) they seem to do relatively solid survey-based work.

brentsch 6y ago

The paper may not offer insightful prescriptions for experienced engineers, but can work like this still be useful for informing future studies in a meaningful way? The authors repeatedly note the widespread inadequacies of the current research landscape. (To anyone familiar with the literature, is their assessment accurate?) In my eyes, the message is that the paper represents an incremental step in the direction of a truly detailed understanding of the factors involved in developer productivity. Even if there's a broad intersection between the answers one would get from taxi drivers and from computer scientists, the distance between the two fields makes that an unexpected result, which should prompt us to change how we think about computer science* and/or how we think about studying it. *or cab driving

bdowling 6y ago
> The authors repeatedly note the widespread inadequacies of the current research landscape. This is standard language in academic research papers. It is there to sell the importance of the research to the reader, in particular to journal editors or peers who review the article. It is mere puffery.
- deleted 6y ago
  [deleted]

texasbigdata 6y ago

In grad school we had a forced module on information systems in HR in which one case was about RBC and their new formerly-Bain Director of HR Data and the 3 or 4 year effort to “quantify” labor. Cases should be taken with some skepticism but let’s assume the case was correct. I wanted to tear my hair out. Not only was the output or ultimate deliverable of this gargantuan activity an infographic of largely non falsifiable random words (trustworthy, reliable, predictable, etc etc), I did a quick back of the envelope that the approach to repeatedly solicit feedback on every employee from every business line and every customer they intersect could generate 5 trillion outbound survey “friction” onto their customers and stakeholders in a given year give or take 2 orders of magnitude. What does that cost in terms of non-enjoyment? Does it even work, and at what response rate at that scale? And what does it cost to maintain, since surely the Canadian treasury at least sort of requires or at worst encourages banks to employ trustworthy employees. Worse, how implementable are the outcome and/insights? For a real open requisition is evaluating “trustworthiness” at time of hire possible? Is it possible to compare two candidates on a relative or absolute basis on this metric? And if they are hired, going forward how is that tracked? More surveys? Perhaps the case wasn’t as bad, and I’m sure the OP HB post of this paper has subtleties in survey design or methods I don’t appreciate, but just to pick on them by cherry picking the first few attributes that stick out: passionate, systemic, data driven, focused, hardworking, persevering, etc?!?!? Worse, where’s the actual data? Like when Google apparently decided to get rid of middle management it felt like a reversible scientific experiment with measurable outcomes and metrics. This doesn’t. For example, from personal experience, in investment banking being hard working especially at the very lowest and newest ranks is similarly lauded and required and emphasized. But with a fairly simple app that tracks second level activity of which app is open and when and can assign a rule a rule based score, when reviewing a few years of data, the data contradicted the folsky saying. The data showed above XX hours per week for me was non-restorative, and further above XX hours would create some incremental inefficiency increase, and above even that was a “tech debt” like factor which later required recuperation (which could also be calculated but maybe that was an accident). In English: if (simple numbers) working 10 hours leads to 8 hours of work and 2 hours of goof off for some productivity, maybe working 14 hours is 10/4, and maybe working 17 is 11/6 and also requires 2 hours of rest at a subsequent trailing period. Depending on what’s going on, maybe it makes sense to do that, maybe it doesn’t. The software engineer view would be much more interesting. It would be fascinating to compare even arbitrary metrics of consenting individuals across a large pool. Who gets the biggest bonus relative to hours worked? What time do peak performers come in and leave at and are there any wfh days? Even trivial but theoretically transferable insights would help: “what percent of top performers drink no caffeine or tea” or “pair programming and velocity”. Heck, buzzfeed baiting analysis like “do top engineers type faster? A 10 part slideshow on why you should switch to a mechanical Dvorak keyboard” or “Can you type XYZ words per minute? Find out if you are dragging your team down”. Anything but more lengthy repetitions of “trust” and “respect” and whatever the word of the day is.

sokoloff 6y ago

One of the key takeaways from the paper for me is the relative ranking of those 54 attributes. If the employees ranked “long-term thinking” above “creates a safe haven”, I have possibly learned something. If engineers rank attributes differently than managers or if cashiers rank attributes differently than tech workers... Side note: I also thought the negative phrasing methodology used was interesting. They asked “is it possible to be a great engineer without this quality?” which I think is most tuned towards qualifying and ranking attributes. Will it tell you individually how to become a great engineer? No, there are hundreds of posts, many on medium.com for that. This paper may not be perfect, but I found a lot to consider inside and “beyond useless” is pretty far from my judgment.