4 ms·

COVID does not constitute a natural experiment! A useful natural experiment is when something changes in a way that can only (or at least almost entirely) affe

COVID does not constitute a natural experiment! A useful natural experiment is when something changes in a way that can only (or at least almost entirely) affect your outcome by way of the cause you're interested in. In the cause and effect chain of A -> B -> C, if A affects B, but not C, then you can exploit wild things in A to learn about how B affects C. But if A affects C outside of B, you can't tell the effects apart. COVID does not constitute a natural experiment! It's a mantra that apparently needs repeating. The pandemic changed work from home. Work from home presumable changes productivity. The pandemic also changes productivity, so you can't tell where the pandemic-driven productivity changes are from more granularly. Learning anything that generalizes requires tremendous care.

ineptech 5y ago
Agreed! The elephants in the room for me are the self-reported hours (Did "hours worked" really increase, or was that time spent changing laundry and running errands that got classified as work time) and the obvious confounder of children being at home rather than in school.
- majormajor 5y ago
  > (Did "hours worked" really increase, or was that time spent changing laundry and running errands that got classified as work time) Let's keep in mind here too that "hours between commute to/from office" != "hours working." Coffee shop runs, lunches, snack breaks, chatting, walking around...
  - munk-a 5y ago
    It's really unfortunate that modern productivity metrics try and maximize things like butt-in-chair hours and time spent actively typing. I work in a rather creative part of CS and those coffee runs and walks are pretty productively spent time for me. One of my issues early on into the pandemic is that I was fearful of taking walks and my productivity actually dropped as my physical exercise decreased. When I realized this and started emphasizing active movement while working (dancing to techno) and taking walks to get in bursts of fresh air I found that things more than reversed. I would note that I have to deal with ADD and physical activity has a long history of positive impacts on folks with such - so that advice might not go for everyone.
- tdeck 5y ago
  Keep in mind also that this is a consulting company, where people are always encouraged to report more hours in order to bill the client.
  - SketchySeaBeast 5y ago
    That's kind of bizarre - I've worked in a couple of consulting gigs now and the hours have always been tightly reigned in - the company wanted all the hours for sure, but they wanted you producing things constantly, just throwing more hours at something and going over estimates was a way to get yourself in trouble quick.
    - akiselev 5y ago
      You're talking about the interface between the worker and the employer, the GP is talking about the interface between employer and client. The employer doesn't want the worker slacking off but that doesn't mean that they won't create situations that result in more billable work, such as by taking an inefficient path under the cover of intangible factors. The irony is optimizing for productivity in one while optimizing for billing in the other.
- cortesoft 5y ago
  I think that doing household chores and errands for supplies SHOULD count a work time if you are working from home. If you are WFH, then you are not only doing your main job, but you are also working in the facilities department. If you go to the office, there are people who clean the office, take out the garbage, maintain equipment, clean the bathroom, etc. At home you have to do all of that.
  - munk-a 5y ago
    You make a decent argument but... society doesn't judge that commute times or medical appointments should necessarily be reimbursed, so I think that there isn't really a "sane baseline" to compare against in non-covid times. There's also the very real question of division of usage - most of us are likely working in spaces that intersect heavily with our private spaces (since we didn't move to our current residences with having a dedicated home office in mind) and that creates a pretty complex process to actually assign responsibility - it shouldn't be 100% in either direction, but it's hard to figure out how much of that time was legitimately caused by you working from home.
jfengel 5y ago
That's true, but it's a pretty common interpretation of the term "natural experiment". Few, if any, "natural experiments" are actually experiments in a rigorous sense. But they are a thing that happens: you have an uncontrolled happenstance that allows you to infer, but not prove, something you want to learn more about. When the surface meaning of a term has no referents, it will often be co-opted to refer to something people need a term for (assuming it's compact and convenient). Another example that comes to mind is "homophobia", which doesn't refer to actual "fear of homosexuals" but rather to a dislike for them -- a thing which is very common but doesn't have a convenient, short expression. (The term actually comes from being the opposite of "homophilia", a reasonably straightforward but outdated term for homosexuality.) So it's true that we don't really have a good control and will have difficulty separating out real causes and effects. But a thing definitely happened and we can definitely try to learn something from it -- a notion that happens often enough to merit a name. If you've got a better one we can try to make it catch on.
- andersource 5y ago
  You're right, but then giving the title "Work from Home & Productivity" is misleading, giving the (IMO false) impression that results are generalizable for WFH beyond COVID. And there are some pretty strong claims: > Therefore, productivity fell by about 20% > Employees also spent less time networking, and received less coaching and 1:1 meetings with supervisors > Employees with children living at home increased hours worked more than those without children at home, and suffered a bigger decline in productivity than those without children I would say these results have more to do with COVID effects and forced, sudden WFH than inherent characteristics of WFH.
  - ghaff 5y ago
    > Employees also spent less time networking, and received less coaching and 1:1 meetings with supervisors And, in normal times, remote workers often have/should have off-sites/on-sites, in-person team meetings, etc. And I haven't actually had fewer 1:1's. Probably more because I haven't canceled as many due to travel. (Which isn't necessarily a positive thing but still.)
  - 5y ago
addicted 5y ago
That’s not what is meant by a natural experiment. https://en.m.wikipedia.org/wiki/Natural_experiment https://en.m.wikipedia.org/wiki/Natural_experiment
pjungwir 5y ago
That seems like a good point. I've worked from home for 15 years, and my productivity (and billable hours) took a big hit last spring, just from the distraction of trying to learn what I could and be prepared. Subjectively I'd say my focus suffered as well. I imagine businesses have lost productivity in all kinds of ways the last ~15 months.
- kurttheviking 5y ago
  I need to read the article but your comment speaks to exactly the analysis I'd like to see: what was the productivity change among those with a stable WFH situation prior to COVID vs. during COVID, and how does that correlate with family status. Personally, the biggest change for me in COVID WFH was not the remote meetings -- I already had plenty of those -- but it was the introduction of a second job: part-time Kindergarten teacher.
1vuio0pswjnm7 5y ago
https://en.wikipedia.org/wiki/Confounding_variable https://en.wikipedia.org/wiki/Confounding_variable
malshe 5y ago
Just curious, who said it is a natural experiment? Certainly not the authors

ineptech 5y ago

Agreed! The elephants in the room for me are the self-reported hours (Did "hours worked" really increase, or was that time spent changing laundry and running errands that got classified as work time) and the obvious confounder of children being at home rather than in school.

majormajor 5y ago
> (Did "hours worked" really increase, or was that time spent changing laundry and running errands that got classified as work time) Let's keep in mind here too that "hours between commute to/from office" != "hours working." Coffee shop runs, lunches, snack breaks, chatting, walking around...
- munk-a 5y ago
  It's really unfortunate that modern productivity metrics try and maximize things like butt-in-chair hours and time spent actively typing. I work in a rather creative part of CS and those coffee runs and walks are pretty productively spent time for me. One of my issues early on into the pandemic is that I was fearful of taking walks and my productivity actually dropped as my physical exercise decreased. When I realized this and started emphasizing active movement while working (dancing to techno) and taking walks to get in bursts of fresh air I found that things more than reversed. I would note that I have to deal with ADD and physical activity has a long history of positive impacts on folks with such - so that advice might not go for everyone.
tdeck 5y ago
Keep in mind also that this is a consulting company, where people are always encouraged to report more hours in order to bill the client.
- SketchySeaBeast 5y ago
  That's kind of bizarre - I've worked in a couple of consulting gigs now and the hours have always been tightly reigned in - the company wanted all the hours for sure, but they wanted you producing things constantly, just throwing more hours at something and going over estimates was a way to get yourself in trouble quick.
  - akiselev 5y ago
    You're talking about the interface between the worker and the employer, the GP is talking about the interface between employer and client. The employer doesn't want the worker slacking off but that doesn't mean that they won't create situations that result in more billable work, such as by taking an inefficient path under the cover of intangible factors. The irony is optimizing for productivity in one while optimizing for billing in the other.
cortesoft 5y ago
I think that doing household chores and errands for supplies SHOULD count a work time if you are working from home. If you are WFH, then you are not only doing your main job, but you are also working in the facilities department. If you go to the office, there are people who clean the office, take out the garbage, maintain equipment, clean the bathroom, etc. At home you have to do all of that.
- munk-a 5y ago
  You make a decent argument but... society doesn't judge that commute times or medical appointments should necessarily be reimbursed, so I think that there isn't really a "sane baseline" to compare against in non-covid times. There's also the very real question of division of usage - most of us are likely working in spaces that intersect heavily with our private spaces (since we didn't move to our current residences with having a dedicated home office in mind) and that creates a pretty complex process to actually assign responsibility - it shouldn't be 100% in either direction, but it's hard to figure out how much of that time was legitimately caused by you working from home.

jfengel 5y ago

That's true, but it's a pretty common interpretation of the term "natural experiment". Few, if any, "natural experiments" are actually experiments in a rigorous sense. But they are a thing that happens: you have an uncontrolled happenstance that allows you to infer, but not prove, something you want to learn more about. When the surface meaning of a term has no referents, it will often be co-opted to refer to something people need a term for (assuming it's compact and convenient). Another example that comes to mind is "homophobia", which doesn't refer to actual "fear of homosexuals" but rather to a dislike for them -- a thing which is very common but doesn't have a convenient, short expression. (The term actually comes from being the opposite of "homophilia", a reasonably straightforward but outdated term for homosexuality.) So it's true that we don't really have a good control and will have difficulty separating out real causes and effects. But a thing definitely happened and we can definitely try to learn something from it -- a notion that happens often enough to merit a name. If you've got a better one we can try to make it catch on.

andersource 5y ago
You're right, but then giving the title "Work from Home & Productivity" is misleading, giving the (IMO false) impression that results are generalizable for WFH beyond COVID. And there are some pretty strong claims: > Therefore, productivity fell by about 20% > Employees also spent less time networking, and received less coaching and 1:1 meetings with supervisors > Employees with children living at home increased hours worked more than those without children at home, and suffered a bigger decline in productivity than those without children I would say these results have more to do with COVID effects and forced, sudden WFH than inherent characteristics of WFH.
- ghaff 5y ago
  > Employees also spent less time networking, and received less coaching and 1:1 meetings with supervisors And, in normal times, remote workers often have/should have off-sites/on-sites, in-person team meetings, etc. And I haven't actually had fewer 1:1's. Probably more because I haven't canceled as many due to travel. (Which isn't necessarily a positive thing but still.)
- 5y ago

addicted 5y ago

That’s not what is meant by a natural experiment. https://en.m.wikipedia.org/wiki/Natural_experiment https://en.m.wikipedia.org/wiki/Natural_experiment

pjungwir 5y ago

That seems like a good point. I've worked from home for 15 years, and my productivity (and billable hours) took a big hit last spring, just from the distraction of trying to learn what I could and be prepared. Subjectively I'd say my focus suffered as well. I imagine businesses have lost productivity in all kinds of ways the last ~15 months.

kurttheviking 5y ago
I need to read the article but your comment speaks to exactly the analysis I'd like to see: what was the productivity change among those with a stable WFH situation prior to COVID vs. during COVID, and how does that correlate with family status. Personally, the biggest change for me in COVID WFH was not the remote meetings -- I already had plenty of those -- but it was the introduction of a second job: part-time Kindergarten teacher.

1vuio0pswjnm7 5y ago

https://en.wikipedia.org/wiki/Confounding_variable https://en.wikipedia.org/wiki/Confounding_variable

malshe 5y ago

Just curious, who said it is a natural experiment? Certainly not the authors