7 ms·

Link to actual study: http://chrisparnin.me/pdf/stress_FSE_20.pdf http://chrisparnin.me/pdf/stress_FSE_20.pdf The study participants were split into two groups

Link to actual study: http://chrisparnin.me/pdf/stress_FSE_20.pdf http://chrisparnin.me/pdf/stress_FSE_20.pdf The study participants were split into two groups. One group solved the problem alone, in silence. The other group was instructed to narrate their thought process out loud to an interviewer who stood over their shoulder (literally, see the photo on page 3) while they solved it. The flaw in this study is that the two groups weren't really assigned the same task. Solving a problem isn't the same as verbally narrating your solution to a problem. The authors attributed the differences not to the extra work narration work assigned to the second group, but to anxiety levels. The participants even explained that their difficulty came from having to talk while solving: > Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”. There are other concerns with the study format such as many of a significant number of participants simply giving up well before the clock ran out, in both the private and public interview groups. They were given 30 minutes to solve 3 problems, but some of the participants (including in the private group) were giving up around the 10 minute mark without solving anything at all. This suggests a very high variance of the underlying abilities of their candidates, which necessitates a much larger sample size to draw conclusions.

deleted 5y ago
[deleted]
kerblang 5y ago
Am I reading this incorrectly or did literally all the female candidates finish the test in private while none of them finished in public? (disclaimer: Am male but still cannot talk and hack at the same time)
- deleted 5y ago
  [deleted]
- Jensson 5y ago
  Also the effect basically disappears if you take only male candidates, so the study is a bit suspect. Men and women typically aren't that different in studies.
  - kerblang 5y ago
    > Men and women typically aren't that different in studies. That's an extremely broad assumption
    - Jensson 5y ago
      I have read a lot of psychological studies, the effect size difference between men and women in the same study is typically pretty small. But in this one the effect is close to 0 for men and 100% for women, that is really suspicious. I wouldn't trust results like these unless they were replicated with more robust methods.
- Manuel_D 5y ago
  Bear in mind there's a sample size of four women.
  - Jensson 5y ago
    There are 10 women. 4 in the private sample and 6 in the public sample.
omarhaneef 5y ago
That would be a flaw in the sense that anxiety may not be the right word, but it does capture the parallel with white board interviews and underscore the problem that you may be turning away people who can perform the job well but not code while explaining.
mrfusion 5y ago
Does it matter if it’s anxiety or not?
- im3w1l 5y ago
  Yes. If it's anxiety you can do things known to reduce anxiety. E.g. taking a beta blocker such as propranolol.
  - Mountain_Skies 5y ago
    Now we expect people to get drugged up just to attend a job interview?
- tonyedgecombe 5y ago
  I think so, unless you expect your workplace to generate the same levels of anxiety during day to day work.
SomeCallMeTim 5y ago
It sounds like the study points to the idea that "talk through your reasoning" is a bad interview approach. The last interview sequence I needed to do they put me in front of an online IDE and said "go code." They also said I could use my OWN IDE if I had it set up. One even said "you can finish it later (post-interview) if you want." The candidates giving up: Some nontrivial percentage of CS students are just ... not capable developers. That's why we use programming tests to begin with. Programming tests may be the worst possible way to assess candidates, but they also seem to be the only way to assess candidates that doesn't involve paying them for a trial period, which really only works for companies large enough to be able to waste a lot of money. "Democracy is the worst form of government, except for all the others."
- mprovost 5y ago
  This is a great point about CS grads. Maybe in the past, you could just take a degree at face value: "oh you graduated from Stanford with an A average, you must know how to code, here's a job". But nobody trusts any university to produce capable developers, so you have to test everyone with crazy interviews. So then what's the point of going to a top CS school and why can they keep charging so much tuition?
  - jedmeyers 5y ago
    > why can they keep charging so much tuition Because everyone keeps paying so much tuition. Why does _everyone_ can afford to pay so much is another question worth exploring.
  - nowherebeen 5y ago
    Because like it or not, people pay for branding. It’s an ego thing or a correlation thing (in the past).
    - gmadsen 5y ago
      I doubt there exists a 4.0 gpa standford cs grad that is incapable of coding. It is more than just branding. Standford is harder and you learn more skills that a run of the mill state school. MIT moreso, no one is skating under the radar at MIT
sharps_xp 5y ago
> Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”. This is why I built myself a tool to practice talking and narrating my thoughts! https://enumerable.co https://enumerable.co
IshKebab 5y ago
> The flaw in this study is that the two groups weren't really assigned the same task. The real flaw is that the title doesn't follow from the research. "Tech sector job interviews assess anxiety, not software skills." That's saying that you're really measuring level of anxiety, not skill. So skill is irrelevant and you're just measuring how anxious people are. But the study didn't show that at all. It just showed that on average, people perform better when they're not anxious. Not exactly a surprising result! Of course people are going to be more anxious in an interview. But if it affects everyone equally who cares? Just make your questions easier.
- tonyedgecombe 5y ago
  I would assume it doesn’t affect people equally. Then the question is whether or not the anxiety will become a problem in the workplace. I can see arguments for both sides.
b20000 5y ago
maybe you should get in touch with the professor and tell him? curious what the answer will be. i think it’s reasonable to think that this study is relevant and that there is truth in it. there is also the issue that every interviewer i encountered assumes lying and incompetence and will conduct the interview in such a way that the result is what they want to see - the candidate failing to write the code within a few minutes. they are not capable of letting a candidate do the interview without biasing the interview and results. this also comes from the automatic assumption that because they work in a FAANG they are a fucking genius and are the only ones who can write software. i can see this happening as they are so brainwashed and lost all touch with reality and how the rest of the industry works. meanwhile there are some marketing execs at that FAANG with a compensation package far above what they will ever get paid.

deleted 5y ago

[deleted]

kerblang 5y ago

Am I reading this incorrectly or did literally all the female candidates finish the test in private while none of them finished in public? (disclaimer: Am male but still cannot talk and hack at the same time)

deleted 5y ago
[deleted]
Jensson 5y ago
Also the effect basically disappears if you take only male candidates, so the study is a bit suspect. Men and women typically aren't that different in studies.
- kerblang 5y ago
  > Men and women typically aren't that different in studies. That's an extremely broad assumption
  - Jensson 5y ago
    I have read a lot of psychological studies, the effect size difference between men and women in the same study is typically pretty small. But in this one the effect is close to 0 for men and 100% for women, that is really suspicious. I wouldn't trust results like these unless they were replicated with more robust methods.
Manuel_D 5y ago
Bear in mind there's a sample size of four women.
- Jensson 5y ago
  There are 10 women. 4 in the private sample and 6 in the public sample.

omarhaneef 5y ago

That would be a flaw in the sense that anxiety may not be the right word, but it does capture the parallel with white board interviews and underscore the problem that you may be turning away people who can perform the job well but not code while explaining.

mrfusion 5y ago

Does it matter if it’s anxiety or not?

im3w1l 5y ago
Yes. If it's anxiety you can do things known to reduce anxiety. E.g. taking a beta blocker such as propranolol.
- Mountain_Skies 5y ago
  Now we expect people to get drugged up just to attend a job interview?
tonyedgecombe 5y ago
I think so, unless you expect your workplace to generate the same levels of anxiety during day to day work.

SomeCallMeTim 5y ago

It sounds like the study points to the idea that "talk through your reasoning" is a bad interview approach. The last interview sequence I needed to do they put me in front of an online IDE and said "go code." They also said I could use my OWN IDE if I had it set up. One even said "you can finish it later (post-interview) if you want." The candidates giving up: Some nontrivial percentage of CS students are just ... not capable developers. That's why we use programming tests to begin with. Programming tests may be the worst possible way to assess candidates, but they also seem to be the only way to assess candidates that doesn't involve paying them for a trial period, which really only works for companies large enough to be able to waste a lot of money. "Democracy is the worst form of government, except for all the others."

mprovost 5y ago
This is a great point about CS grads. Maybe in the past, you could just take a degree at face value: "oh you graduated from Stanford with an A average, you must know how to code, here's a job". But nobody trusts any university to produce capable developers, so you have to test everyone with crazy interviews. So then what's the point of going to a top CS school and why can they keep charging so much tuition?
- jedmeyers 5y ago
  > why can they keep charging so much tuition Because everyone keeps paying so much tuition. Why does _everyone_ can afford to pay so much is another question worth exploring.
- nowherebeen 5y ago
  Because like it or not, people pay for branding. It’s an ego thing or a correlation thing (in the past).
  - gmadsen 5y ago
    I doubt there exists a 4.0 gpa standford cs grad that is incapable of coding. It is more than just branding. Standford is harder and you learn more skills that a run of the mill state school. MIT moreso, no one is skating under the radar at MIT

sharps_xp 5y ago

> Participants also had difficulty with performing tasks that involved multiple simultaneous actions. Participants felt stressed by having to “talk while trying to write” (P44), “talking while writing” (P25), and “think and talk and do code at the same time” (P39). P41 found it difficult to “constantly speak during solving” and “lost breath at a few places during the task”. This is why I built myself a tool to practice talking and narrating my thoughts! https://enumerable.co https://enumerable.co

IshKebab 5y ago

> The flaw in this study is that the two groups weren't really assigned the same task. The real flaw is that the title doesn't follow from the research. "Tech sector job interviews assess anxiety, not software skills." That's saying that you're really measuring level of anxiety, not skill. So skill is irrelevant and you're just measuring how anxious people are. But the study didn't show that at all. It just showed that on average, people perform better when they're not anxious. Not exactly a surprising result! Of course people are going to be more anxious in an interview. But if it affects everyone equally who cares? Just make your questions easier.

tonyedgecombe 5y ago
I would assume it doesn’t affect people equally. Then the question is whether or not the anxiety will become a problem in the workplace. I can see arguments for both sides.

b20000 5y ago

maybe you should get in touch with the professor and tell him? curious what the answer will be. i think it’s reasonable to think that this study is relevant and that there is truth in it. there is also the issue that every interviewer i encountered assumes lying and incompetence and will conduct the interview in such a way that the result is what they want to see - the candidate failing to write the code within a few minutes. they are not capable of letting a candidate do the interview without biasing the interview and results. this also comes from the automatic assumption that because they work in a FAANG they are a fucking genius and are the only ones who can write software. i can see this happening as they are so brainwashed and lost all touch with reality and how the rest of the industry works. meanwhile there are some marketing execs at that FAANG with a compensation package far above what they will ever get paid.