11 ms·

I shudder at the data reliability issues of a self-selecting sample of individuals being asked to self-report salaries. There may well be real pay inequity iss

I shudder at the data reliability issues of a self-selecting sample of individuals being asked to self-report salaries. There may well be real pay inequity issues -- in fact, my default presumption is that such problems exist anywhere that there has not been a systematic effort to avoid them -- but unless there's far more to this spreadsheet than the article describes, I don't see how it can reasonably be taken to have any large-scale significance.

geofft 11y ago
I think there's as much a story in the managerial response to the spreadsheet as in the spreadsheet itself. If there really was not a problem (or not a statistically-significant one, or not a statistically-significantly worse-than-industry-norms one), it seems like there would have been better ways to respond. Even "HR did the stats, you can't see them, but our data implies less wage gap than your data" would be worth something. And regardless of whether we can infer anything about the true pay inequity from the sample, the response by management described here is just totally awful, and strongly implies behavior that is illegal in the US. You're not allowed to penalize an employee for discussing wages simply because their discussion is factually inaccurate or statistically unsound.
- cperciva 11y ago
  Oh, I agree that the managerial response (particularly with her getting flooded by bonuses which her manager vetoed) was idiotic. That doesn't take away from my concerns about how people may be analysing and using the anecdata though. If someone came to me and said "I did a self-selecting self-reporting survey of salaries, and it says I'm underpaid", I think I'd be more likely to fire them for incompetence than to give them a raise.
  - anu_gupta 11y ago
    An astonishing attitude. Perhaps, if you ever end up managing people, you'll hopefully take a step back and contemplate investigating whether there was, in fact, a valid concern.
  - asgard1024 11y ago
    > If someone came to me and said "I did a self-selecting self-reporting survey of salaries, and it says I'm underpaid", I think I'd be more likely to fire them for incompetence than to give them a raise. Even if besides them being wrong in this one thing, they would actually be competent and underpaid? If the answer to the above question is no, why do you even have to elaborate. Then it doesn't actually matter at all if the employee complains rightly or wrongly! I have to conclude the answer has to be yes, then. It's perfectly understandable that you, as a manager or employer, don't want to pay people for their merit if you don't have to. But it's not something you should be proud of.
    - cperciva 11y ago
      Drawing conclusions from self-selected self-reported data (without a careful process for validation and debiasing) is a big red flag for (a) being ignorant of basic statistics, and (b) not even being aware of your ignorance of basic statistics. I mean, "why do we use random samples?" is covered in the first lecture of a 1st year stats course. I expect developers to be familiar with basic statistics.
      - dlitz 11y ago
        Wow. Colin, are you just trying to win an argument online or are you really that flippant when it comes to firing an employee? Keep in mind that we're talking about people's livelihoods, here---it's the sort of thing that can stress people out to the point of mental breakdown, even if you're not actively threatening them. Firstly, this isn't a job interview. If someone's been working with you for long enough that they want to discuss a salary adjustment, then shouldn't your assessment of the value that they bring to the table be more informed than looking for "red flags" on the spot? Secondly, developers are generally not familiar with basic statistics. If it's important to your business that they be better-than-average at stats, then budget some time and money for training. Thirdly, in the real world, we make decisions from incomplete and biased information all the time. Sometimes, those decisions turn out to be wrong, but on the whole, it still works out better than pretending that we have no information at all. If you're in management and you have access to more complete information, then the smart thing for an employee to do is to have a conversation with you. It makes no sense to fire someone for that. I have a lot of respect for your technical work, but the management style you're describing so far sounds authoritarian and abusive. I really hope that's not how you'd actually treat your reports, in practice.
  - jsmthrowaway 11y ago
    In this thread, the repeatedly reinforced observation that expertise in crafting software does not automatically bestow skills for managing, understanding, or empathizing with people, and those whose work you respect tremendously may just disappoint you when it comes to working for them.
  - deleted 11y ago
    [deleted]
  - DannyBee 11y ago
    Vetoing > 1 peer bonus for the same thing is pretty much policy. Certainly it's manager discretion, but that's usually the right thing to do. (There are other forms of bonusing that would make more sense in such a case, like a spot bonus, or whatever)
  - geofft 11y ago
    > If someone came to me and said "I did a self-selecting self-reporting survey of salaries, and it says I'm underpaid", I think I'd be more likely to fire them for incompetence than to give them a raise. OK, except that's not what's happening here. What seems to be happening here is someone saying "I did a self-reporting survey of salaries, and there's an existence proof that someone at my level is getting paid $X, can I get paid closer to $X than my current salary." Which is sound. The statistics and pivot tables are very interesting, but I suspect they're not directly being used for negotiations. Nobody has any good information about how much money they should be asking for. The data this spreadsheet provides is that, if you ask for $X, it's within the range of reason, and not so high that you'll get laughed at. If I apply for Google and try to negotiate a $200K salary, is that low? high? so high I'll get my offer revoked? (I've heard of that happening in tech!) I have no idea. Knowing even one person who works in my future role and makes $200K would be valuable information. So would knowing that I can't find anyone in that role making more than $110K. Knowing the mean and standard deviation of salaries at 95% confidence is not particularly more useful to me.
    - cperciva 11y ago
      What seems to be happening here is someone saying "I did a self-reporting survey of salaries, and there's an existence proof that someone at my level is getting paid $X, can I get paid closer to $X than my current salary." Which is sound. It's sound modulo the assumption that everybody is being honest when they report their salaries, sure. Of course, in a company Google's size, there are always going to be outliers, so "someone at my level is getting paid $X" is a long way from "everybody at my level should be getting paid $X". If I apply for Google and try to negotiate a $200K salary, is that low? high? so high I'll get my offer revoked? (I've heard of that happening in tech!) I have no idea. Right, and that's the proper use of such anecdata -- to help you formulate your bargaining strategy, not as a bargaining chip itself. Ironically, if information like this had been available back in 2006, I might be working for Google: A few months after I rejected their offer, I was told that it was "pathetically low" and that I could have negotiated at least 50% more. If I had been offered that much more it might have factored into my decision; but I didn't know I could negotiate, so I ended up simply rejecting the offer.
georgemcbay 11y ago
Well, the obvious solution to the self-selecting issue is just have the company publish all salaries publicly. Of course, very few would dare to do that because they know that the data will be hiding all sorts of embarrassing and possibly resignation-causing truths. Which is kind of shitty if you think about it, especially when taken with the notion that "companies are people"... sociopathic, lying, cheating people...
- icebraining 11y ago
  On the other hand, if I was a manager at an established company with thousands of workers, I'd be very wary of "just publishing all salaries publicly" even if I did think they were all fair (which I don't believe it's true for most companies, granted). For one thing, fairness is subjective. For many, "equal pay for equal work" is fair, but many others believe that seniority, or a better CV, or many others factors should be rewarded differently. Secondly, there are issues of perception. A worker might think (s)he should be paid more because (s)he always leaves later than his/her colleague, without taking into account that the colleague prefers to take shorter lunches or fewer breaks and leave earlier. You can explain it to them, but why would they believe you? It'd be different if a company published the salaries from the start, but switching after growing to a huge size can be very disruptive, and I don't blame managers for avoiding opening that can of worms. (Again, I don't believe the salaries at Google are perfectly fair, I'm talking about an hypothetical company)
  - mtrimpe 11y ago
    Then again; in all of Sweden everybody's tax returns are public information and their economy still seems to do just fine...
    - icebraining 11y ago
      Yes, but that's been open for over 100 years. What I was talking about was the issues surrounding the transition from closed to open, not the openness itself.
- 11y ago
mrow84 11y ago
Statistics is as much (if not more) about careful thought as it is about applying standard tests. What sampling biases do you think might be present in this sampling that invalidate the stated conclusions? For example, for a conclusion about gender differences to be exaggerated by the data there would have to be systematic biases to sampling women with lower salaries, and men with higher salaries, in the same position. Does this seem likely?
- walshemj 11y ago
  Ah the want do you want 2+2 to equal boss :-)
- cperciva 11y ago
  For a conclusion about gender differences to be exaggerated by the data there would have to be systematic biases to sampling women with lower salaries, and men with higher salaries, in the same position. Does this seem likely? For a self-selected sample, absolutely. Self-worth is far more tied to salary in males than in females. (I don't think this is innate; I think it's a result of social conditioning which tells males that their role in society is to be breadwinners.) But you don't need to be able to point to a specific source of bias to say that self-selected samples are problematic. It's up to the person who intends to use the statistics to show that they were either gathered reliably or carefully examined to remove biases.
  - mrow84 11y ago
    I don't understand the link you're drawing between self-worth and willingness to participate in this spreadsheet exercise, but I suppose it is beside the point. I certainly agree that if someone is trying to make a statistical claim then they should be the ones to do the due diligence on the data, and on their analyses, to show that they stand up - I would certainly expect this in a more formal publication. What I am disputing is that one can dismiss the conclusions simply due to the lack of these checks, without considering what the data might tell you. Nearly always such dismissive arguments strike me as a disagreement with the conclusions, rather than a claim about failings in the data - a more convincing argument would present a reason why the data are flawed, such as the one you have given in this response (even if I don't understand it).

geofft 11y ago

I think there's as much a story in the managerial response to the spreadsheet as in the spreadsheet itself. If there really was not a problem (or not a statistically-significant one, or not a statistically-significantly worse-than-industry-norms one), it seems like there would have been better ways to respond. Even "HR did the stats, you can't see them, but our data implies less wage gap than your data" would be worth something. And regardless of whether we can infer anything about the true pay inequity from the sample, the response by management described here is just totally awful, and strongly implies behavior that is illegal in the US. You're not allowed to penalize an employee for discussing wages simply because their discussion is factually inaccurate or statistically unsound.

cperciva 11y ago
Oh, I agree that the managerial response (particularly with her getting flooded by bonuses which her manager vetoed) was idiotic. That doesn't take away from my concerns about how people may be analysing and using the anecdata though. If someone came to me and said "I did a self-selecting self-reporting survey of salaries, and it says I'm underpaid", I think I'd be more likely to fire them for incompetence than to give them a raise.
- anu_gupta 11y ago
  An astonishing attitude. Perhaps, if you ever end up managing people, you'll hopefully take a step back and contemplate investigating whether there was, in fact, a valid concern.
- asgard1024 11y ago
  > If someone came to me and said "I did a self-selecting self-reporting survey of salaries, and it says I'm underpaid", I think I'd be more likely to fire them for incompetence than to give them a raise. Even if besides them being wrong in this one thing, they would actually be competent and underpaid? If the answer to the above question is no, why do you even have to elaborate. Then it doesn't actually matter at all if the employee complains rightly or wrongly! I have to conclude the answer has to be yes, then. It's perfectly understandable that you, as a manager or employer, don't want to pay people for their merit if you don't have to. But it's not something you should be proud of.
  - cperciva 11y ago
    Drawing conclusions from self-selected self-reported data (without a careful process for validation and debiasing) is a big red flag for (a) being ignorant of basic statistics, and (b) not even being aware of your ignorance of basic statistics. I mean, "why do we use random samples?" is covered in the first lecture of a 1st year stats course. I expect developers to be familiar with basic statistics.
    - dlitz 11y ago
      Wow. Colin, are you just trying to win an argument online or are you really that flippant when it comes to firing an employee? Keep in mind that we're talking about people's livelihoods, here---it's the sort of thing that can stress people out to the point of mental breakdown, even if you're not actively threatening them. Firstly, this isn't a job interview. If someone's been working with you for long enough that they want to discuss a salary adjustment, then shouldn't your assessment of the value that they bring to the table be more informed than looking for "red flags" on the spot? Secondly, developers are generally not familiar with basic statistics. If it's important to your business that they be better-than-average at stats, then budget some time and money for training. Thirdly, in the real world, we make decisions from incomplete and biased information all the time. Sometimes, those decisions turn out to be wrong, but on the whole, it still works out better than pretending that we have no information at all. If you're in management and you have access to more complete information, then the smart thing for an employee to do is to have a conversation with you. It makes no sense to fire someone for that. I have a lot of respect for your technical work, but the management style you're describing so far sounds authoritarian and abusive. I really hope that's not how you'd actually treat your reports, in practice.
- jsmthrowaway 11y ago
  In this thread, the repeatedly reinforced observation that expertise in crafting software does not automatically bestow skills for managing, understanding, or empathizing with people, and those whose work you respect tremendously may just disappoint you when it comes to working for them.
- deleted 11y ago
  [deleted]
- DannyBee 11y ago
  Vetoing > 1 peer bonus for the same thing is pretty much policy. Certainly it's manager discretion, but that's usually the right thing to do. (There are other forms of bonusing that would make more sense in such a case, like a spot bonus, or whatever)
- geofft 11y ago
  > If someone came to me and said "I did a self-selecting self-reporting survey of salaries, and it says I'm underpaid", I think I'd be more likely to fire them for incompetence than to give them a raise. OK, except that's not what's happening here. What seems to be happening here is someone saying "I did a self-reporting survey of salaries, and there's an existence proof that someone at my level is getting paid $X, can I get paid closer to $X than my current salary." Which is sound. The statistics and pivot tables are very interesting, but I suspect they're not directly being used for negotiations. Nobody has any good information about how much money they should be asking for. The data this spreadsheet provides is that, if you ask for $X, it's within the range of reason, and not so high that you'll get laughed at. If I apply for Google and try to negotiate a $200K salary, is that low? high? so high I'll get my offer revoked? (I've heard of that happening in tech!) I have no idea. Knowing even one person who works in my future role and makes $200K would be valuable information. So would knowing that I can't find anyone in that role making more than $110K. Knowing the mean and standard deviation of salaries at 95% confidence is not particularly more useful to me.
  - cperciva 11y ago
    What seems to be happening here is someone saying "I did a self-reporting survey of salaries, and there's an existence proof that someone at my level is getting paid $X, can I get paid closer to $X than my current salary." Which is sound. It's sound modulo the assumption that everybody is being honest when they report their salaries, sure. Of course, in a company Google's size, there are always going to be outliers, so "someone at my level is getting paid $X" is a long way from "everybody at my level should be getting paid $X". If I apply for Google and try to negotiate a $200K salary, is that low? high? so high I'll get my offer revoked? (I've heard of that happening in tech!) I have no idea. Right, and that's the proper use of such anecdata -- to help you formulate your bargaining strategy, not as a bargaining chip itself. Ironically, if information like this had been available back in 2006, I might be working for Google: A few months after I rejected their offer, I was told that it was "pathetically low" and that I could have negotiated at least 50% more. If I had been offered that much more it might have factored into my decision; but I didn't know I could negotiate, so I ended up simply rejecting the offer.

georgemcbay 11y ago

Well, the obvious solution to the self-selecting issue is just have the company publish all salaries publicly. Of course, very few would dare to do that because they know that the data will be hiding all sorts of embarrassing and possibly resignation-causing truths. Which is kind of shitty if you think about it, especially when taken with the notion that "companies are people"... sociopathic, lying, cheating people...

icebraining 11y ago
On the other hand, if I was a manager at an established company with thousands of workers, I'd be very wary of "just publishing all salaries publicly" even if I did think they were all fair (which I don't believe it's true for most companies, granted). For one thing, fairness is subjective. For many, "equal pay for equal work" is fair, but many others believe that seniority, or a better CV, or many others factors should be rewarded differently. Secondly, there are issues of perception. A worker might think (s)he should be paid more because (s)he always leaves later than his/her colleague, without taking into account that the colleague prefers to take shorter lunches or fewer breaks and leave earlier. You can explain it to them, but why would they believe you? It'd be different if a company published the salaries from the start, but switching after growing to a huge size can be very disruptive, and I don't blame managers for avoiding opening that can of worms. (Again, I don't believe the salaries at Google are perfectly fair, I'm talking about an hypothetical company)
- mtrimpe 11y ago
  Then again; in all of Sweden everybody's tax returns are public information and their economy still seems to do just fine...
  - icebraining 11y ago
    Yes, but that's been open for over 100 years. What I was talking about was the issues surrounding the transition from closed to open, not the openness itself.
11y ago

mrow84 11y ago

Statistics is as much (if not more) about careful thought as it is about applying standard tests. What sampling biases do you think might be present in this sampling that invalidate the stated conclusions? For example, for a conclusion about gender differences to be exaggerated by the data there would have to be systematic biases to sampling women with lower salaries, and men with higher salaries, in the same position. Does this seem likely?

walshemj 11y ago
Ah the want do you want 2+2 to equal boss :-)
cperciva 11y ago
For a conclusion about gender differences to be exaggerated by the data there would have to be systematic biases to sampling women with lower salaries, and men with higher salaries, in the same position. Does this seem likely? For a self-selected sample, absolutely. Self-worth is far more tied to salary in males than in females. (I don't think this is innate; I think it's a result of social conditioning which tells males that their role in society is to be breadwinners.) But you don't need to be able to point to a specific source of bias to say that self-selected samples are problematic. It's up to the person who intends to use the statistics to show that they were either gathered reliably or carefully examined to remove biases.
- mrow84 11y ago
  I don't understand the link you're drawing between self-worth and willingness to participate in this spreadsheet exercise, but I suppose it is beside the point. I certainly agree that if someone is trying to make a statistical claim then they should be the ones to do the due diligence on the data, and on their analyses, to show that they stand up - I would certainly expect this in a more formal publication. What I am disputing is that one can dismiss the conclusions simply due to the lack of these checks, without considering what the data might tell you. Nearly always such dismissive arguments strike me as a disagreement with the conclusions, rather than a claim about failings in the data - a more convincing argument would present a reason why the data are flawed, such as the one you have given in this response (even if I don't understand it).