4 ms·

By and large, GlassDoor ratings are no longer a good measure of how well a company treats employees; they now measure, mainly, whether a company has the ability

by cs702 8y ago

By and large, GlassDoor ratings are no longer a good measure of how well a company treats employees; they now measure, mainly, whether a company has the ability to engineer and maintain artificially good GlassDoor ratings. GlassDoor, in short, has become a textbook example of Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."[a] The same phenomenon is known in some contexts as Campbell's Law: "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."[b] [a] https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law [b] https://en.wikipedia.org/wiki/Campbell%27s_law https://en.wikipedia.org/wiki/Campbell%27s_law

DamnInteresting 8y ago
> a textbook example of Goodhart's Law I had never heard of Goodhart's Law by name until about 45 minutes ago, while listening to an episode of Planet Money that originally aired last November. And here it is again. This is one of the more striking examples of the Baader-Meinhof phenomenon I've experienced in recent years.
- d23 8y ago
  I just saw it in another thread today and am seeing it again here. It's pretty apropos for what I'm dealing with at the company I'm at right now too.
- shlomok 8y ago
  Once I learned about the Baader-Meinhof phenomenon it seemed to pop up everywhere.
  - tatami 8y ago
    It's spreading on HN this past week, you could say it's viral.
mushufasa 8y ago
You are correct that companies' direct incentive is to optimize their ratings. Don't you think that's correlated with improving actual employee experiences, though? I agree they're not the same thing. But if they spend HR effort to improve the ratings, that means they're looking at the feedback, and presumably they will try to address some or part of it where possible. Essentially, the fact that people optimize metrics doesn't mean that the whole process is hopeless. Anyone who has built metrics systems will agree that you can rarely measure the exact thing you're looking for -- the ideal is to get as close as you can.
- jacques_chester 8y ago
  > But if they spend HR effort to improve the ratings, that means they're looking at the feedback, and presumably they will try to address some or part of it where possible. No. In any control loop you have two ways of reducing error. One is the arduous business of actuation. Doing things. Soul-searching. Working hard. Investigating. Listening. Making it better. The other is to lie to the sensor. This is always easier. From the controller's POV, it's indistinguishable.
  - mushufasa 8y ago
    Don't both ways happen? Surely you don't see the world as completely black-and-white?
    - jacques_chester 8y ago
      I said it's easier to deceive the sensor. The way you avoid this problem is to avoid creating the incentive to subvert the loop. I am fairly confident that every metric, target, index, goal, KPI or score that has ever been connected to money, power or fame has been subverted. As far as that goes, yes, I see the world as black-and-white. Glassdoor scores are tied to ability to hire. It's easier to game the scores than to not suck, especially since "horrible place to work" is going to correlate strongly with "dubious ethics".
DoctorOetker 8y ago
sorry, this is off topic, but it's impossible to add a comment to the discussion in which you participated at https://news.ycombinator.com/item?id=18364148 https://news.ycombinator.com/item?id=18364148 where you discuss with one of the authors of the excellent paper "Towards Understanding Linear Analogies" you are discussing ELMo and specifically word senses i.e. "leaves" which has multiple senses (departs, foliage, ...) I recently stumbled on a paper from 2016 (modified 2018) which IMHO gives a lot of insight, but I had to read both the old and the new version (I recommend reading v1 first and then the newest) They illustrate how for example the word "leave" in word embeddings, is in fact simply a linear combination (with coefficients on the order of 1) of the true positions of each individual sense i.e. "leave" = A"leave.1"+B"leave.2"+... with A,B, ... constants close to 1. Theres typically less than 10 for a single word. These reside in the same vector space as the word embeddings, and they illustrate how these sense vectors can be retrieved from the shallow word embedding vectors by sparse coding! The paper is at https://arxiv.org/abs/1601.03764 https://arxiv.org/abs/1601.03764 Again I recommend reading first v1 then v6
- cs702 8y ago
  Interesting. Based on a quick glance, it seems this would answer the question I asked in that thread about whether it might be possible to get word-sense embeddings via two simpler transformations: first a transformation to the space of word-sense compositions (e.g., via GloVe/SGNS), and then a transformation to the space of word senses. I'll take a closer look. Thank you!
  - DoctorOetker 8y ago
    correct, the flow of information is: corpus -- word2vecOrGloVe--> word embeddings v_w in R^n word embedding --sparsecoding--> sense embeddings v_s in same R^n the sparse coding process gives the constants A_ws and senses where subscript w is a word index and s is a sense index, so that: word vectors v_w = sum(A_ws v_s, s) and for each word w most A_ws are zero except for a few s values 1) polysemy: a word w can have multiple senses, namely those sense vectors with index s where A_ws is nonzero 2) synonyms: a sense s can have multiple synonyms w, again those w where A_ws is nonzero so the result of sparse coding gives for each word, a couple of indexes of the sense vectors, and for each sense the corresponding indexes of word vectors... and of course the sense vectors themselves. so that to find say a synonym of "leaves", you just look at the sense indexes corresponding to that string, then you look at the different words indexes for that sense, and they will refer to the words "foliage" but also "leaves" of course and possibly others... I also believe that once you have the sense vectors, in theory a second pass through the corpus should improve results if the context of each focus word is used to determine the closest sense vector compatible with the focus word... so that in effect word2vec or Glove extraction is run on the senses instead of the words for the sparse coding they used SMALLbox, and I am still trying to better grasp how exactly the sparse coding works, and what prevents the A_ws and v_s to reduce to the trivial solution v_s = v_w and A_ww = 1 and A_wx = 0 for x differing from w...

by cs702 8y ago

DamnInteresting 8y ago

> a textbook example of Goodhart's Law I had never heard of Goodhart's Law by name until about 45 minutes ago, while listening to an episode of Planet Money that originally aired last November. And here it is again. This is one of the more striking examples of the Baader-Meinhof phenomenon I've experienced in recent years.

d23 8y ago
I just saw it in another thread today and am seeing it again here. It's pretty apropos for what I'm dealing with at the company I'm at right now too.
shlomok 8y ago
Once I learned about the Baader-Meinhof phenomenon it seemed to pop up everywhere.
- tatami 8y ago
  It's spreading on HN this past week, you could say it's viral.

mushufasa 8y ago

You are correct that companies' direct incentive is to optimize their ratings. Don't you think that's correlated with improving actual employee experiences, though? I agree they're not the same thing. But if they spend HR effort to improve the ratings, that means they're looking at the feedback, and presumably they will try to address some or part of it where possible. Essentially, the fact that people optimize metrics doesn't mean that the whole process is hopeless. Anyone who has built metrics systems will agree that you can rarely measure the exact thing you're looking for -- the ideal is to get as close as you can.

jacques_chester 8y ago
> But if they spend HR effort to improve the ratings, that means they're looking at the feedback, and presumably they will try to address some or part of it where possible. No. In any control loop you have two ways of reducing error. One is the arduous business of actuation. Doing things. Soul-searching. Working hard. Investigating. Listening. Making it better. The other is to lie to the sensor. This is always easier. From the controller's POV, it's indistinguishable.
- mushufasa 8y ago
  Don't both ways happen? Surely you don't see the world as completely black-and-white?
  - jacques_chester 8y ago
    I said it's easier to deceive the sensor. The way you avoid this problem is to avoid creating the incentive to subvert the loop. I am fairly confident that every metric, target, index, goal, KPI or score that has ever been connected to money, power or fame has been subverted. As far as that goes, yes, I see the world as black-and-white. Glassdoor scores are tied to ability to hire. It's easier to game the scores than to not suck, especially since "horrible place to work" is going to correlate strongly with "dubious ethics".

DoctorOetker 8y ago

sorry, this is off topic, but it's impossible to add a comment to the discussion in which you participated at https://news.ycombinator.com/item?id=18364148 https://news.ycombinator.com/item?id=18364148 where you discuss with one of the authors of the excellent paper "Towards Understanding Linear Analogies" you are discussing ELMo and specifically word senses i.e. "leaves" which has multiple senses (departs, foliage, ...) I recently stumbled on a paper from 2016 (modified 2018) which IMHO gives a lot of insight, but I had to read both the old and the new version (I recommend reading v1 first and then the newest) They illustrate how for example the word "leave" in word embeddings, is in fact simply a linear combination (with coefficients on the order of 1) of the true positions of each individual sense i.e. "leave" = A"leave.1"+B"leave.2"+... with A,B, ... constants close to 1. Theres typically less than 10 for a single word. These reside in the same vector space as the word embeddings, and they illustrate how these sense vectors can be retrieved from the shallow word embedding vectors by sparse coding! The paper is at https://arxiv.org/abs/1601.03764 https://arxiv.org/abs/1601.03764 Again I recommend reading first v1 then v6

cs702 8y ago
Interesting. Based on a quick glance, it seems this would answer the question I asked in that thread about whether it might be possible to get word-sense embeddings via two simpler transformations: first a transformation to the space of word-sense compositions (e.g., via GloVe/SGNS), and then a transformation to the space of word senses. I'll take a closer look. Thank you!
- DoctorOetker 8y ago
  correct, the flow of information is: corpus -- word2vecOrGloVe--> word embeddings v_w in R^n word embedding --sparsecoding--> sense embeddings v_s in same R^n the sparse coding process gives the constants A_ws and senses where subscript w is a word index and s is a sense index, so that: word vectors v_w = sum(A_ws v_s, s) and for each word w most A_ws are zero except for a few s values 1) polysemy: a word w can have multiple senses, namely those sense vectors with index s where A_ws is nonzero 2) synonyms: a sense s can have multiple synonyms w, again those w where A_ws is nonzero so the result of sparse coding gives for each word, a couple of indexes of the sense vectors, and for each sense the corresponding indexes of word vectors... and of course the sense vectors themselves. so that to find say a synonym of "leaves", you just look at the sense indexes corresponding to that string, then you look at the different words indexes for that sense, and they will refer to the words "foliage" but also "leaves" of course and possibly others... I also believe that once you have the sense vectors, in theory a second pass through the corpus should improve results if the context of each focus word is used to determine the closest sense vector compatible with the focus word... so that in effect word2vec or Glove extraction is run on the senses instead of the words for the sparse coding they used SMALLbox, and I am still trying to better grasp how exactly the sparse coding works, and what prevents the A_ws and v_s to reduce to the trivial solution v_s = v_w and A_ww = 1 and A_wx = 0 for x differing from w...