9 ms·

This is such a great example of bullshit studies. Yeah I'm gonna go on a limb and say this is a marketing campaign. It's an insult to science to call this a "s

This is such a great example of bullshit studies. Yeah I'm gonna go on a limb and say this is a marketing campaign. It's an insult to science to call this a "study". From the pdf: > The fieldwork of this study was carried out at remarkable speed, from the 23rd to 24th of June 2021 Data was collected in 1 day. Is this a joke? No it's not! > The population sampled included Software Engineers aged 18+ living in the United Kingdom. In total we surveyed 258 such people Survation, the market research agency who did the actual work, interviewed 258 people only, I don't see any definition or criteria on what that means, years of experience, field, cultural-background, or any of the many other very important aspects that anyone would wonder about. I really don't understand how they can do a half-assed data survey on 258 people only, and then have the gall to claim "83% of developers". What's really sad about this is that there is indeed an increased workload I've heard people say that they've experienced, caused by the side-effects of the covid-response most governments took, from working from home for extended periods to mental health issues. This kind of click-bait bullshit marketing campaigns is pretty harmful to the very real issue of burnout because it muddles the ground and dilutes the issue.

himoacs 5y ago
Thanks for taking the time to read the details. I am always skeptical when I see "Study finds...".
jquery 5y ago
I’m guilty of posting before reading the article. I’m glad you pointed this out. Sounds like an awful study. There’s true burnout and then there’s feelings of stress. They aren’t the same thing at all. I know because I’ve experienced both. Burnout cuts you down to your soul. I haven’t programmed professionally in 2 years and the thought of doing so makes me nauseous. That’s burnout.
- systemvoltage 5y ago
  Let's me 100% honest here - if we cannot define clearly the psychological and medical definition of a burn-out (with accurate diagnosis), it is just fine to discuss in non-scientific context. The moment you call this a study and no one here is willing to ask hard questions makes me extremely disappointed in HN and the intellectual curiosity that usually drives discussions here. Why do I feel like there is some political angle to this? (labor rights).
  - smoldesu 5y ago
    It would take a lot of mental gymnastics to twist this into a political statement. If we "cannot define clearly the psychological and medical definition of a burn-out", why are we even studying it in the first place? That would be like experimenting without a control group.
- jquery 5y ago
  Looking back on this comment, I want to apologize for trying to say what is "real" burnout and what isn't. Thinking about it more, they can all be symptoms of the same thing, to varying degrees. Apologies to anyone I may have upset by minimizing their experience.
pmoriarty 5y ago
"Data was collected in 1 day. Is this a joke? No it's not!" What makes you think data collected over 1 day worse than the same data collected in multiple days? "I really don't understand how they can do a half-assed data survey on 258 people only, and then have the gall to claim "83% of DEVELOPERS"" It's called extrapolation. No study ever interviews everyone. They all have to extrapolate from a subset of the entire population. Obviously the more participants there are the more confidence you can have in the conclusion, but whether 200, 2000, or 20000 participants would make for a "good" study is completely arbitrary.
- jedimastert 5y ago
  > They all have to extrapolate from a subset of the entire population. This only really works if you have a representative subset of he population you're extrapolating to. Is the distribution of experience roughly equivalent to the general population of "developers"? Age distribution? Company size? # of children? Home life? These kinds of demographics can have a big impact, and I see no discussion of how they were accounted for. Heck, even the general work ethic culture difference between the UK vs the rest of the world could be a huge confounding factor that I know wasn't really accounted for.
  - antris 5y ago
    Getting a perfectly representative subset of the population, means surveying the whole population of the world. Also, accounting for every imaginable variable is always impossible, because we cannot ethically grow people in an isolated environment. Sure, a meta-analysis of several studies reveals more than one single study, but that doesn't make a single study bad or worthless - or as said in the parent - "a bullshit study".
    - jedimastert 5y ago
      There are plenty of ways to account for not having a representative sample distribution, I'm just saying that I didn't see any kind of discussion about demographics or confounding variables in the actual study. This was a pilot study, a way to open an avenue of discussion, and to be sure I think the study warrants more investigation. They even say so: "In future we would like to conduct larger studies over multiple countries."
roody15 5y ago
Sadly lots of today’s “research” meets this low threshold. Lots of politicized headlines use low quality research to push narratives. #depressing
- marvindanig 5y ago
  I recently came across a US Government funded study [1] that used anecdata of 12 people–just 12 tired souls!– to put forth a seal on the claim that e-ink screens are better for the human eyes than emissive type of displays. Edit: Reality however: the opposite is true. Only in some scenarios (like reading under open sun) does one type of screen outdo the other. In most usual cases, that is indoor reading, the emissive display is better for the eyes. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5929099/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5929099/
mirekrusin 5y ago
If they asked me to do this survey, their mail would end up in my spam box. I’m not burned out.
p_j_w 5y ago
Your criticism of the sample size is unfounded. You absolutely can have meaningful results with only 258 people provided you have good sampling. If you want to criticize their sampling, then by all means do it, but that's not what you've done here.
- enraged_camel 5y ago
  Sorry, but what part of "they only surveyed software developers in the United Kingdom" gives you the impression that their sampling might actually be good? Worth noting that the study itself doesn't seem to mention their sampling methodology.
  - antris 5y ago
    Studies are never meant to be an ultimate answer to everything. And just because they're not the ultimate answer, doesn't mean that they are therefore worthless, bad or "bullshit".
    - danenania 5y ago
      It actually does mean that in most cases. Having false confidence in a hypothesis based on non-rigorous studies is much worse (and in some cases, a lot more dangerous) than withholding judgment until the data (all the data, not some cherry-picked slice of it) really is conclusive.
    - enraged_camel 5y ago
      No. You perform studies to validate observations or theories and, if you are able to validate them, offer potential factors as contributing causes. A study where the data does not support the conclusion is worthless, and can in fact be harmful.
    - myfavoritedog 5y ago
      I know this refers to actual scientific studies, but the same garbage-in, garbage-out problem applies. If anything, meaningless surveys like this are even less reliable. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778
dheera 5y ago
The thing is, everything in this world is now gauged by engagement, not quality. The fact that this is getting more reactions on HN gives it even more exposure, and actively reinforces this behavior even more. If you have a really good, thorough study that is so thorough that people have nothing to comment on it, it will not rise to the top of pretty much any platform, because comments are valued higher than upvotes on most platforms. So you want to engineer some clickbait in there so that your article gets madly commented upon; being incomplete is perhaps the easiest way to do this. (I don't know about HN, but it certainly appears Facebook and Twitter work that way.)
- throwawayswede 5y ago
  > everything in this world is now gauged by engagement nope not true, only algorithm driven website. just because this post or comment made it to the top of HN doesn't mean people think it's valuable now, it just means like you said: it has buzz around it, and it should, because it's bullshit and should be exposed for the bullshit it is. Is that enough to get people to be more skeptical about what they read? I don't know, probably not though.
myWindoonn 5y ago
Your arguments are specious; perhaps you're simply angry that you didn't understand an industry you've worked in for a while? Perhaps you're in management, or in the 20% who aren't burnt out, and you're embarrassed by proxy? Psychological experiments and surveys are often done in only one or two days. Why would a wider window of time help when taking a snapshot of a current situation? Hm, is 258 a small sample size? Let's put in some effort instead of just screaming "bullshit" a bunch. There are approximately 408,000 people in the entire population [0], and plugging everything into a calculator, we get two results: First, they should have tried to poll around 400 people (and perhaps they did!), but more importantly, the 95% confidence interval is only around ±6%. Let me repeat that more clearly: By typical polling standards (95% confidence interval, random sample from filtered population), we expect that 77-89% of UK software developers are actually burnt out. That is still three out of every four people in the industry! Not sure what to say. Take a stats class? [0] https://www.statista.com/statistics/318818/numbers-of-programmers-and-software-development-professionals-in-the-uk/ https://www.statista.com/statistics/318818/numbers-of-progra...
- Lutger 5y ago
  It's a lot and I'm not surprised. Recent research in my country (from a reputable institute, the Netherlands) mentions 1 out of 6 people feel burnout and ranks IT industry in the top 3 sectors. This survey articulates it in quite a broad way however, just asking if respondents have 'feelings of burnout', which most of us have once in a while. That doesn't mean they are burned out clinically speaking, or even if they have symptoms of burnout. It does point to a large, systematic problem in the industry (and society in general).
- throwawayswede 5y ago
  Your comment starts with a disingenuous sentence, and continues to show a level of bitterness I've seen before in people who believe themselves to be more knowledgeable/experienced than their surroundings (or they actually are), but never got a chance (or so they believe). I didn't claim to know everything there is to know about statistics and research in general, but I think I know enough to tell that this is not a study, but a marketing campaign for whatever this usehaystack thing is. > First, they should have tried to poll around 400 people (and perhaps they did!), If you had read the article, you would have known that that's not what they did. > By typical polling standards (95% confidence interval, random sample from filtered population), we expect that 77-89% of UK software developers are actually burnt out. That is still three out of every four people in the industry! Do I understand correctly that you've worked on this? That certainly explains the hostility, but nevertheless. > we expect that 77-89% of UK software developers are actually burnt out. That's not what the link/study says. > That is still three out of every four people in the industry! This is absolutely wrong, without a shred of a doubt. I recommend that you read: How to lie with Statistics by Darrell Huff.
alienthrowaway 5y ago
I wouldn't be so quick to discard it. From personal experience, the past 14+ months of working from home has been very stressful. I didn't think I was burnt out, until I started interviewing (due to work "stress", which turned out to be burn-out in disguise). One recruiter informed me they have seen a spike - 70% of interviewees canceled scheduled interviews, post-screening. I almost cancelled mine due to being overwhelmed/burnout from preparations + work, and my thesis is there are a lot of burned out developers out there. Unfortunately, I didn't hear what the cancellation rate baseline was before work-from-home. If there are tech recruiters/HR folk/hiring managers reading this, I'd love to know the numbers you've been seeing for interview cancelations.
- SirZimzim 5y ago
  There should be a study on studies being posted to HN. We must be at like 99% of them that get critiqued for being a bad study in one way or another. I would love to see some examples where everyone accepts the study.
  - LolWolf 5y ago
    To be extremely fair to hn: almost all such studies about human behavior, especially those relying on polling, are garbage. Often, the statistical methods are inappropriate at best, or downright deceiving at worst; there is nominal, if any, understanding about what "controlling for a variable" means, among many other such things. Not to mention that often the methods themselves are highly questionable in their reproducibility, statistics aside. (Another classic is to point out flaws in generalizations for mice studies and such, but that is, imo, independent of the study's methods. It's a fair critique, but I don't think it's what you're interested in measuring w.r.t. "bad" studies.)
mikehearn 5y ago
Having a poll in the field for two days is pretty common, and n=258 is definitely enough for a representative sample.
- throwawayswede 5y ago
  - This was not a poll. - Being common doesn't mean it's good or acceptable. In this case, it doesn't account for the effects of day of the week, or what sort of issues the person was having on that day specifically. - n=258 is enough if it was sampled properly, and they provide no details to show that it was, so it wasn't. They provided nothing more than the engineers they surveyed were in the UK. Nothing on experience, age group distribution, family-size, years of work, company size, work change. - As it stands, this provides 0 valuable information, and I can only assume it's intended for buzz.
  - mikehearn 5y ago
    > "It doesn't account for... what sort of issues the person was having on that day specifically" I'm not clear on what you are suggesting here. Are you saying they should contact people multiple times to make sure they weren't having a bad day the first time around? 2-3 days is common because any longer than that and the survey results will no longer be reflective of a specific point in time. Also, if you check their crosstabs they have breakdowns for gender and age.
stcredzero 5y ago
This is such a great example of bullshit studies. Lots of social science seems to be confirmation of stuff that people have been suspecting. Not to take away from such things: It's important to verify with data. However, the really interesting question: Where does the burnout come from? Is it possible to specifically characterize the source? By starting to break it down like that, we start toward finding a solution.
ottermax8 5y ago
Yeah I can do a survey too :)

himoacs 5y ago

Thanks for taking the time to read the details. I am always skeptical when I see "Study finds...".

jquery 5y ago

I’m guilty of posting before reading the article. I’m glad you pointed this out. Sounds like an awful study. There’s true burnout and then there’s feelings of stress. They aren’t the same thing at all. I know because I’ve experienced both. Burnout cuts you down to your soul. I haven’t programmed professionally in 2 years and the thought of doing so makes me nauseous. That’s burnout.

systemvoltage 5y ago
Let's me 100% honest here - if we cannot define clearly the psychological and medical definition of a burn-out (with accurate diagnosis), it is just fine to discuss in non-scientific context. The moment you call this a study and no one here is willing to ask hard questions makes me extremely disappointed in HN and the intellectual curiosity that usually drives discussions here. Why do I feel like there is some political angle to this? (labor rights).
- smoldesu 5y ago
  It would take a lot of mental gymnastics to twist this into a political statement. If we "cannot define clearly the psychological and medical definition of a burn-out", why are we even studying it in the first place? That would be like experimenting without a control group.
jquery 5y ago
Looking back on this comment, I want to apologize for trying to say what is "real" burnout and what isn't. Thinking about it more, they can all be symptoms of the same thing, to varying degrees. Apologies to anyone I may have upset by minimizing their experience.

pmoriarty 5y ago

"Data was collected in 1 day. Is this a joke? No it's not!" What makes you think data collected over 1 day worse than the same data collected in multiple days? "I really don't understand how they can do a half-assed data survey on 258 people only, and then have the gall to claim "83% of DEVELOPERS"" It's called extrapolation. No study ever interviews everyone. They all have to extrapolate from a subset of the entire population. Obviously the more participants there are the more confidence you can have in the conclusion, but whether 200, 2000, or 20000 participants would make for a "good" study is completely arbitrary.

jedimastert 5y ago
> They all have to extrapolate from a subset of the entire population. This only really works if you have a representative subset of he population you're extrapolating to. Is the distribution of experience roughly equivalent to the general population of "developers"? Age distribution? Company size? # of children? Home life? These kinds of demographics can have a big impact, and I see no discussion of how they were accounted for. Heck, even the general work ethic culture difference between the UK vs the rest of the world could be a huge confounding factor that I know wasn't really accounted for.
- antris 5y ago
  Getting a perfectly representative subset of the population, means surveying the whole population of the world. Also, accounting for every imaginable variable is always impossible, because we cannot ethically grow people in an isolated environment. Sure, a meta-analysis of several studies reveals more than one single study, but that doesn't make a single study bad or worthless - or as said in the parent - "a bullshit study".
  - jedimastert 5y ago
    There are plenty of ways to account for not having a representative sample distribution, I'm just saying that I didn't see any kind of discussion about demographics or confounding variables in the actual study. This was a pilot study, a way to open an avenue of discussion, and to be sure I think the study warrants more investigation. They even say so: "In future we would like to conduct larger studies over multiple countries."

roody15 5y ago

Sadly lots of today’s “research” meets this low threshold. Lots of politicized headlines use low quality research to push narratives. #depressing

marvindanig 5y ago
I recently came across a US Government funded study [1] that used anecdata of 12 people–just 12 tired souls!– to put forth a seal on the claim that e-ink screens are better for the human eyes than emissive type of displays. Edit: Reality however: the opposite is true. Only in some scenarios (like reading under open sun) does one type of screen outdo the other. In most usual cases, that is indoor reading, the emissive display is better for the eyes. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5929099/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5929099/

mirekrusin 5y ago

If they asked me to do this survey, their mail would end up in my spam box. I’m not burned out.

p_j_w 5y ago

Your criticism of the sample size is unfounded. You absolutely can have meaningful results with only 258 people provided you have good sampling. If you want to criticize their sampling, then by all means do it, but that's not what you've done here.

enraged_camel 5y ago
Sorry, but what part of "they only surveyed software developers in the United Kingdom" gives you the impression that their sampling might actually be good? Worth noting that the study itself doesn't seem to mention their sampling methodology.
- antris 5y ago
  Studies are never meant to be an ultimate answer to everything. And just because they're not the ultimate answer, doesn't mean that they are therefore worthless, bad or "bullshit".
  - danenania 5y ago
    It actually does mean that in most cases. Having false confidence in a hypothesis based on non-rigorous studies is much worse (and in some cases, a lot more dangerous) than withholding judgment until the data (all the data, not some cherry-picked slice of it) really is conclusive.
  - enraged_camel 5y ago
    No. You perform studies to validate observations or theories and, if you are able to validate them, offer potential factors as contributing causes. A study where the data does not support the conclusion is worthless, and can in fact be harmful.
  - myfavoritedog 5y ago
    I know this refers to actual scientific studies, but the same garbage-in, garbage-out problem applies. If anything, meaningless surveys like this are even less reliable. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778

dheera 5y ago

The thing is, everything in this world is now gauged by engagement, not quality. The fact that this is getting more reactions on HN gives it even more exposure, and actively reinforces this behavior even more. If you have a really good, thorough study that is so thorough that people have nothing to comment on it, it will not rise to the top of pretty much any platform, because comments are valued higher than upvotes on most platforms. So you want to engineer some clickbait in there so that your article gets madly commented upon; being incomplete is perhaps the easiest way to do this. (I don't know about HN, but it certainly appears Facebook and Twitter work that way.)

throwawayswede 5y ago
> everything in this world is now gauged by engagement nope not true, only algorithm driven website. just because this post or comment made it to the top of HN doesn't mean people think it's valuable now, it just means like you said: it has buzz around it, and it should, because it's bullshit and should be exposed for the bullshit it is. Is that enough to get people to be more skeptical about what they read? I don't know, probably not though.

myWindoonn 5y ago

Your arguments are specious; perhaps you're simply angry that you didn't understand an industry you've worked in for a while? Perhaps you're in management, or in the 20% who aren't burnt out, and you're embarrassed by proxy? Psychological experiments and surveys are often done in only one or two days. Why would a wider window of time help when taking a snapshot of a current situation? Hm, is 258 a small sample size? Let's put in some effort instead of just screaming "bullshit" a bunch. There are approximately 408,000 people in the entire population [0], and plugging everything into a calculator, we get two results: First, they should have tried to poll around 400 people (and perhaps they did!), but more importantly, the 95% confidence interval is only around ±6%. Let me repeat that more clearly: By typical polling standards (95% confidence interval, random sample from filtered population), we expect that 77-89% of UK software developers are actually burnt out. That is still three out of every four people in the industry! Not sure what to say. Take a stats class? [0] https://www.statista.com/statistics/318818/numbers-of-programmers-and-software-development-professionals-in-the-uk/ https://www.statista.com/statistics/318818/numbers-of-progra...

Lutger 5y ago
It's a lot and I'm not surprised. Recent research in my country (from a reputable institute, the Netherlands) mentions 1 out of 6 people feel burnout and ranks IT industry in the top 3 sectors. This survey articulates it in quite a broad way however, just asking if respondents have 'feelings of burnout', which most of us have once in a while. That doesn't mean they are burned out clinically speaking, or even if they have symptoms of burnout. It does point to a large, systematic problem in the industry (and society in general).
throwawayswede 5y ago
Your comment starts with a disingenuous sentence, and continues to show a level of bitterness I've seen before in people who believe themselves to be more knowledgeable/experienced than their surroundings (or they actually are), but never got a chance (or so they believe). I didn't claim to know everything there is to know about statistics and research in general, but I think I know enough to tell that this is not a study, but a marketing campaign for whatever this usehaystack thing is. > First, they should have tried to poll around 400 people (and perhaps they did!), If you had read the article, you would have known that that's not what they did. > By typical polling standards (95% confidence interval, random sample from filtered population), we expect that 77-89% of UK software developers are actually burnt out. That is still three out of every four people in the industry! Do I understand correctly that you've worked on this? That certainly explains the hostility, but nevertheless. > we expect that 77-89% of UK software developers are actually burnt out. That's not what the link/study says. > That is still three out of every four people in the industry! This is absolutely wrong, without a shred of a doubt. I recommend that you read: How to lie with Statistics by Darrell Huff.

alienthrowaway 5y ago

I wouldn't be so quick to discard it. From personal experience, the past 14+ months of working from home has been very stressful. I didn't think I was burnt out, until I started interviewing (due to work "stress", which turned out to be burn-out in disguise). One recruiter informed me they have seen a spike - 70% of interviewees canceled scheduled interviews, post-screening. I almost cancelled mine due to being overwhelmed/burnout from preparations + work, and my thesis is there are a lot of burned out developers out there. Unfortunately, I didn't hear what the cancellation rate baseline was before work-from-home. If there are tech recruiters/HR folk/hiring managers reading this, I'd love to know the numbers you've been seeing for interview cancelations.

SirZimzim 5y ago
There should be a study on studies being posted to HN. We must be at like 99% of them that get critiqued for being a bad study in one way or another. I would love to see some examples where everyone accepts the study.
- LolWolf 5y ago
  To be extremely fair to hn: almost all such studies about human behavior, especially those relying on polling, are garbage. Often, the statistical methods are inappropriate at best, or downright deceiving at worst; there is nominal, if any, understanding about what "controlling for a variable" means, among many other such things. Not to mention that often the methods themselves are highly questionable in their reproducibility, statistics aside. (Another classic is to point out flaws in generalizations for mice studies and such, but that is, imo, independent of the study's methods. It's a fair critique, but I don't think it's what you're interested in measuring w.r.t. "bad" studies.)

mikehearn 5y ago

Having a poll in the field for two days is pretty common, and n=258 is definitely enough for a representative sample.

throwawayswede 5y ago
- This was not a poll. - Being common doesn't mean it's good or acceptable. In this case, it doesn't account for the effects of day of the week, or what sort of issues the person was having on that day specifically. - n=258 is enough if it was sampled properly, and they provide no details to show that it was, so it wasn't. They provided nothing more than the engineers they surveyed were in the UK. Nothing on experience, age group distribution, family-size, years of work, company size, work change. - As it stands, this provides 0 valuable information, and I can only assume it's intended for buzz.
- mikehearn 5y ago
  > "It doesn't account for... what sort of issues the person was having on that day specifically" I'm not clear on what you are suggesting here. Are you saying they should contact people multiple times to make sure they weren't having a bad day the first time around? 2-3 days is common because any longer than that and the survey results will no longer be reflective of a specific point in time. Also, if you check their crosstabs they have breakdowns for gender and age.

stcredzero 5y ago

This is such a great example of bullshit studies. Lots of social science seems to be confirmation of stuff that people have been suspecting. Not to take away from such things: It's important to verify with data. However, the really interesting question: Where does the burnout come from? Is it possible to specifically characterize the source? By starting to break it down like that, we start toward finding a solution.

ottermax8 5y ago

Yeah I can do a survey too :)