10 ms·

Modern workers are at the mercy of ratings

177 points by edward 7y ago

TurkishPoptart 7y ago
Sadly, this article is paywalled for me.
- emilburzo 7y ago
  https://outline.com/JhCCJd https://outline.com/JhCCJd
Jaygles 7y ago
I get that its immeasurably frustrating for a person to be distilled into performance metrics, but how else can we measure one's impact? Using a trust based method where a supervisor leans on their personal biases to determine an employee's value is fraught with issues as well. A large enough corporation is a machine and we can't base decisions off of how we "feel" the machine's parts are working. Hard numbers will win over time.
- jdc 7y ago
  I think it really comes down to how well-designed the performance model is. And averaging a handful of survey answers sure as hell doesn't cut it.
- Nasrudith 7y ago
  The alternative is management who knows the domain well enough to understand those they supervise. That is expensive and hard in itself to ensure reliably. It is a matter of the push and pull of scale advantages at varied sizes essentially which is essentially universal but varies by domain.
- lazyjeff 7y ago
  Professors have been teaching and getting student course evaluations since forever. At institutions and roles where the admin knows little about the subject, they rely more on the course evals. At institutions with more engaged faculty and with more serious teaching, course evals are less prominent and defer to other forms of assessment (faculty observations, teaching statements, etc.).
  - jld 7y ago
    Historically professors have tenure (admittedly less so now, I believe), so how they score on evaluations has no real bearing on them keeping their jobs. I had a professor who was a terrible lecturer, and he knew his student evaluations were terrible every time. As he handed out the evaluations he would tell us that he did not read them, and did not care what we wrote. I graduated twenty years ago, but just checked the department's website. He's still employed!
    - lazyjeff 7y ago
      Fair enough, but even the tenured professors were untenured at some point before so they did matter and were not problematic enough to deny tenure. And tenure doesn't really apply the same for adjuncts or some lecturers, who are perhaps even more dependent on student evals. And of course it depends on the seriousness the university takes teaching, as there are R1s that are more research oriented and care less about teaching, but they account for 1% of all universities.
      - virtuous_signal 7y ago
        Even as a grad student, my teaching evaluations couldn't negatively affect me. The only people it could have real, negative consequences for were instructors, who work on contracts. As an example, my department head (perhaps unintentionally) sent everyone an email saying any scores of less than 3.73 on a 1-7 scale were grounds for termination of that instructor's contract. I did not envy them at all, basically being highly educated customer service workers.
  - hogFeast 7y ago
    Every large-scale student evaluation I have ever seen has been massively uninformative because students on different courses usually have drastically different expectations. At my university, the law school was top-notch (at least, it is the best in the country) and it got the lowest score by far every year. Why? High expectations, huge number of students competing hard. The best school? Theology. Why? Low expectations, not many students in a far less competitive environment (And I studied in both and in probably four or five faculties across the school...Law was the best, Theology the worst...not even close, the former had people from Law firms, you got full feedback on everything...the latter, you usually got no feedback on any work). This is really the point...now, you can do some kind of grouped mean model or you can compare over time...but what is the actual point? The reason they do this (or did this at my uni) was to compare departments/courses...but everyone knew it was bullshit...people lost jobs (I actually met the Dean who masterminded this, he came from the private sector, taught a minor course in the business school, and was utterly clueless).
- vraivroo 7y ago
  Yes, you've described why large corporations are hated. It's quite obvious that a handful of numbers are incapable of measuring a person's true impact, yet by god, they insist on trying.
  - WalterBright 7y ago
    To be fair, metrics are often insisted upon by unions in order to not allow people to be fired based on the company's feelings about them. They're also insisted upon by courts to adjudicate wrongful termination cases as well as discrimination cases.
  - tcbawo 7y ago
    All businesses want good employees, but what they want most are employees that can be replaced if necessary. They hire people to accomplish tasks and add value. I have felt underappreciated in corporate roles before, but then I realized that I was attaching too much of my identity to my job and in something where I had no real equity. Since then, I am much better at separating work from personal life, and I am happier putting my personal interests above the company's.
    - VRay 7y ago
      Yeah, I was a lot less upset about some of the crappy experiences I'd had working for evil giant companies after I read "The E-Myth Revisited". My elite 10 person team was doing a great job on Device A, while Device B had a team of 200 semi-skilled foreign contractors just barely keeping things on track. Then, somehow, the company decided to reorg my super-skilled team and promoted the leader of the contractor team. From 3-4 levels up in the company, the Device A team and the Device B team were IDENTICAL. They were both delivering things on time and under budget, even though team A was costing the company 1/5th as much, they were both fine. These products were generating hundreds of millions of dollars or maybe even billions in revenue per year, so the difference $5 million/year and $25 million was irrelevant. If I'd realized this sooner, I wouldn't have put so much effort into doing a great job at the expense of my own interests, and it would've been just fine
  - 7y ago
- hhs 7y ago
  > I get that its immeasurably frustrating for a person to be distilled into performance metrics, but how else can we measure one's impact? I think it may also be useful to talk about the reliability and validity of all the things that are measured. Also, how useful is this stuff among the different fields? The article talks about gig workers and manufacturers. Would be interesting if these metrics were applied rigorously to other fields like politicians, talk show pundits, and forecasters.
  - mjfl 7y ago
    This was the basic premise behind the book "Expert political judgement" by Phil Tetlock.
    - hhs 7y ago
      Thanks for the reference.
      - jacques_chester 7y ago
        It's an outstanding read. His followup Superforecasters was co-authored with a journalist for a more popular audience, but still fascinating.
  - onceUponADime 7y ago
    No, no, this rigorious ratings are for lower class endavours. Higher classes get a title, which is a one off rubber stamp of quality and then will never again be judged on performance standing alone. Dr. Senior. Dipl. Ing. Except by there peers, who will lower a silk of omerta over "internal" affairs like a lack of quality in work. Of course, between these professions, there are meta-power struggles, which will play out, in a court- stylized fashion, a eternal ritualized war between castes like "management" and "technical execution". Metrics, as demanded by one caste, are worthless, as they rely on a working interface provided by the oppossition or in depth knowledge, making a allegiance two too factions necessary. Thus all that remains of them is a ritualized insistence, to compensate for the creeping feeling of a lack of controll. For those who need them, can not get them and be certain they are accurate. Making metrics, actually - a flag of truce.
- ceejayoz 7y ago
  > how else can we measure one's impact? This is a case of https://en.wikipedia.org/wiki/Politician%27s_syllogism https://en.wikipedia.org/wiki/Politician%27s_syllogism. We must do something This is something Therefore, we must do this.
  - jschwartzi 7y ago
    Not to be confused with "Do something, even if it's wrong" which is how you break out of analysis paralysis.
  - dredmorbius 7y ago
    A fave. See also: Don't just do something, stand there! https://quoteinvestigator.com/2014/03/22/stand-there/ https://quoteinvestigator.com/2014/03/22/stand-there/
- grawprog 7y ago
  >A large enough corporation is a machine A machine in which the parts of are made up of humans with 'feelings'. Distilling human beings down to cogs in a machine is bad enough when it comes from executives. This belief that corporations are some kind of non-human machine like entity is part of why the world's so fucked today. Maybe the world would be a better place if corporations weren't run as inhuman machines with no care or regard to their employees or the world in which they operate.
  - jiggliemon 7y ago
    > This belief that corporations are some kind of non-human machine like entity is part of why the world's so fucked today. What point in history are you arguing was less fucked than it is today?
    - lstamour 7y ago
      Before corporations were people, in the US at least? http://reclaimdemocracy.org/corporate-accountability-history-corporations-us/ http://reclaimdemocracy.org/corporate-accountability-history... Not at all going to claim it as ideal, but it's a starting point for considering other possible alternatives to today's giant, faceless corporations where nobody within the corporation is easily held accountable, etc.
- 3pt14159 7y ago
  I think the core problem is having one universal metric. Take Uber or Lyft as an example. Some people like silence, some like conversation, some like music. If the ML models were better or if it were based on a web of trust we wouldn't need to have a score that treats a 4.5 as universally better than a 4.6 or similar.
  - deleted 7y ago
    [deleted]
- hogFeast 7y ago
  You should have more than one supervisor (and solicit information from peers). But basically...yes...using judgement, learning how to assess skills...that is all part of management. You cannot turn a bad manager into a good one with information. I have seen no evidence that "hard numbers" work (they aren't "hard"...there is usually almost no justification for a particular metric). And it feeds on innumerable human biases (inability to confront uncertainty, using "hard" information to justify an emotional conclusion, creating meaningless targets, etc.). In my experience, the practical issue is two-fold: one, people are just going to do what they do anyway but they ignore warnings signs/are more unflexible and two, you optimise for data that is obvious, collectable, and (usually) completely irrelevant. This is a trend, it will pass as all these trends do. It started off as quite a good idea (businesses were often failing to set targets or quantify success/failure) and has just got totally out of control.
- moksly 7y ago
  I work in the public sector where the pendulum is always swinging between trust and control. It's partly because we're not entirely driven by MBA's and our success criteria isn't earning shareholders value, but it's mainly because we've been around for so long because we can't go bankrupt when we fail. There are advantages and disadvantage to both approaches. In a system of trust the key benefits are better products, happier employees who have less sick days and stick around longer, more creativity and new solutions that are appreciated by the community. The downside is that some employees, managers or contractors are going to take advantage of the trust in such a terrible way that you'll have no option but to swing towards control once the media finds out. A good example from my country is a decade back, when we had a lot of trust in our public sector and an elderly care facility was saving money by putting dirty diapers back on the elderly if the urine in the diapers didn't weigh enough. Crazy, right? But that's the stuff that happens with trust, you'll also see managers contracting out procurement contracts to companies they own themselves and other stuff like that. On the opposite side you have the MBA based approach where everything is streamlined, measured and conformed. This is where you can build two hospitals for the price of 1,2 because the plan for them are exactly the same. It's where you LEAN every single business process up, and harvest the benefits. It's efficient, clean, and it slowly exhumes your business of value. Because the MBA approach very rarely manages to create something new, it's just really great at maximizing your current system. Eventually a neighboring city is going to do something wild, like figure out how to completely wipe out dyslexia by trying some crazy new method, because they trusted some hippie teacher with a plan. This is where you're forced to swing back toward trust and value tradeskills (not sure if this is the right word in English) because your politicians can't survive another period of having the highest dyslexia numbers in the country and nothing in the MBA playbook can help them. This is a little simplified of course, and I'm certain things are different in the private sector, but I dare say, that if your organisation sticks around for long enough, it's bound to see the pendulum swing between those two outlines.
  - JamesBarney 7y ago
    I think the bigger issue with the MBA approach is it's always possible to trade a little bit of long term hard to measure value for short term profits.
- taurath 7y ago
  Hard numbers have been winning, and thats been the problem. Any time you have a set of numbers to describe a thing, you're creating an imperfect model of that thing. When you only consider that imperfect model, the things moves in the direction of fulfilling that model. You can't say you care about employee well-being, unless you have a metric, or several metrics, it won't ever be done.
- Consultant32452 7y ago
  I'm a bit on the spectrum and I absolutely love having my performance distilled down to a set of metrics. It clearly communicates to me which types of actions are valued in a way that I will often miss from normal social interactions due to my blindness to social queues. The way I perceive work is very helpful in this regard. To me, the purpose of work is to maximize my income/lifestyle while not dipping below a lower bound threshold of misery. I don't seek personal or emotional fulfillment from work at all. I did that when I was younger and, for me, it was a bad way to approach life. This way I'm much happier and my focus/attention is more on the kinds of things I hope will wind up in my obit: family, relationships, charity, etc. I wonder if sales people ever write articles about how they're at the "mercy" of their sales metrics. The idea of it seems hilarious.
  - VRay 7y ago
    It's all well and good until you get judged by a bunch of completely borked-up, unfair metrics. The computer sets you to running halfway across a gigantic warehouse again and again, then measures your "picks per hour" vs somebody who consistently ends up in a tiny corner of the warehouse. Your manager agrees that it's not a problem and tells you not to worry, then a few months down the line some algorithm decides to give your lucky coworker a big raise and you don't get a boost at all. Thankfully I haven't had to deal with any metrics like this for more than a couple of months at a time.. Since engineers' specialty is to optimize ratings, it's an obvious-enough waste of time to apply them that they're usually short-lived.
- CKN23-ARIN 7y ago
  > how else can we measure one's impact? Goodhart's law suggests that this is impossible.
- closeparen 7y ago
  Yet people who passed laboratory science classes in college, people who should really know better, choose to govern by the metrics they "feel" are causally related to outcomes. Maybe you really do have a formula that outperforms a panel of subject matter experts. It does happen sometimes. Not for long, because the experts start using it. But the bar for making this kind of claim is, at minimum, peer-reviewed experimental results that reproduce. Human experience is a rich dataset. Intuition is a sophisticated machine. It blows my mind that people will so easily dismiss all of that in favor of a few bits of information and some 3rd grade arithmetic they just made up. It's not scientifically rigorous just because there's a table and a graph. Come on!
endtime 7y ago
I'd rather be at the mercy of _relatively_ objective metrics than at the mercy of whoever my boss happens to be. (I happen to like my current management chain, but people come and go...)
- decebalus1 7y ago
  Meh.. you're still at the mercy of whoever your boss happens to be. They just need to put a little effort into framing the metrics the right way or to make sure you're getting work that doesn't touch on the right metrics. The idea that performance metrics are fair is just corporate propaganda designed to make you have the incentive to improve them, thus being easier to control and replace.
  - allset_ 7y ago
    >or to make sure you're getting work that doesn't touch on the right metrics. What? The metrics should be distilled from the overall company/org/team goals and mission and if you're working on things that don't align with those of course you're not going to have a good performance review.
    - gravitas 7y ago
      Think more subtly - if you need to meet a metric/goal of 50% of your time on phone calls and 50% on tickets, a manager who wants to tank your metrics will game the system to feed you less than needed to make your metric on one or the other. Just because a metric exists does not mean it's applied fairly or evenly to all employees.
  - endtime 7y ago
    I said "relatively". Of course having a bad relationship with my boss would be bad for my career, no question.
- nitwit005 7y ago
  More generally, I'd prefer being judged by a large number of people rather than one or two managers who may or may not have any idea what's going on. I left my previous team partly because we went through three managers in less than a year, and I rarely saw them. I assume the performance feedback for the team was effectively nonsense.
makotech222 7y ago
This is the unstoppable march of capitalism, turning even its workers into commodities; predicted quite accurately by Marx in the 1800s. Half of our workforce will be able to be completely represented by some number. No different than a bushel of apples.
deleted 7y ago
[deleted]
CiPHPerCoder 7y ago
...does the "0.6 of a worker" bit remind anyone else of the "Three-Fifths compromise" from American History class? https://en.m.wikipedia.org/wiki/Three-Fifths_Compromise https://en.m.wikipedia.org/wiki/Three-Fifths_Compromise Not sure if an intentional dog whistle
WalterBright 7y ago
There's nothing new about this. My first job back in 1979 came with ratings used to determine raises. Corporations are rated, as well, on all sorts of numerical metrics.
greendestiny_re 7y ago
Hacker News should revel in this kind of headline; ratings are made by algorithms that can be gamed and exploited.
maxk42 7y ago
We've always been at the mercy of ratings. Now instead of being rated by a single person, we are rated by hundreds.
virtuous_signal 7y ago
One good thing to come out of certain rating systems, like Uber and Airbnb, is that the users get rated too. If someone is a terrible customer, then they can probably make things a lot worse for the driver or host than vice versa, and without these scores, drivers might not be able to warn other drivers about them. I would hope that with other modern customer service jobs, that reverse-ratings are always employed. It is a measurable way to improve the jobs of customer service workers.
- aprdm 7y ago
  I think there’s a bit of an imbalance as customers can easily create new accounts
  - rock_hard 7y ago
    Not really...most of these services require to register with a phone number creating a pretty high barrier to create additional accounts for the common folk
    - xnyan 7y ago
      If you wanted to rent an Airbnb and you couldn’t because your phone number was associated with a negatively reviewed account, anyone who had the ability to get a phone number has the ability to get a second. It may be a minor inconvenience but not a particularly costly one. If I was a renter and wanted to disassociate from bad reviews, I imagine the easiest way to do it would be to transfer overship of the property to a holding company or similar. This is far harder to do and there’s far more cost.
    - gruez 7y ago
      >most of these services require to register with a phone number creating a pretty high barrier to create additional accounts for the common folk getting a prepaid sim is a "high barrier"?
      - jaxn 7y ago
hechang1997 7y ago
We should start a rating boycott where we rate highest for all ratings.
- aphextron 7y ago
  > We should start a rating boycott where we rate highest for all ratings. This company I've worked at for about a year now is the first place I've been exposed to worker ratings in my career, and this has been my strategy. I refuse to play the game, and give everyone a perfect score no matter what. It absolutely does not benefit us workers in any way to give someone less than perfect marks across the board. It is literally nothing but a tool used by management to provide justification when it comes time to downsize. Furthermore I've noticed that it tends to breed a culture of distrust and backstabbing among colleagues, who may use the opportunity as retribution for personal slights. It's an awful policy.
0xff00ffee 7y ago
A) Paywall. B) Get a different job.
ChainOfFools 7y ago
Les Measurables
jariel 7y ago
They need to do controlled trials whereby they try their best to truly and objectively measure a sample and then compare those to real ratings. I suggest that for most things, customer ratings are completely irrelevant. They might want to ask specific, relevant questions, such as: "Was the car/flat generally clean and tidy" "Were there any problems entering the facility" on the customer side "Did the customer generate problems with the authorities" "Did the customer disrupt your ability to drive" "Did the customer make repeated requests for out-of-bounds services even when they were informed such requests could not be fulfilled" "Did the customer party include more individuals than indicated on the reservation" etc. etc. Then they can glean specific bits of information and provide guidance. Otherwise, general reviews are pointless because they're muddy, set to different standards, emotional, unspecific etc.
kqr 7y ago
I seem to be the first to raise the Deming point of view again, but I'm glad to, so here goes: One of the big problems with trying to rate workers is that most workers have approximately the same rating. In other words, the differences in ratings are not statistically significant; they are just as well explained by pure chance. Building a system where you incentivise and promote on a random draw of cards is bad in and of itself, but doing it and then labeling it as being based on merit is actively hostile to anything you're trying to accomplish. You are building a culture of people stabbing each other in the back in desperation to win your lottery, and not a healthy environment of cooperation and focus on productivity. Yes, sometimes you have a worker who performs off the charts in either direction. That's a great opportunity! If they perform much better than someone else, put them in a position where they can coach and train others in how to do things that well. If they are much worse, try training them or otherwise find them a different job they can do better. Other than that special case, the only way to improve the performance of the system is to give the workers better tools, better information, better understanding, more cooperation, better ability to do their job. The manager is the only one standing in the way of the workers doing a better job. But then how do you assign bonuses and raises when everyone performs approximately the same? Split them across everyone. That simple. Think in terms of incentivising the team, not the individual. And things like promotions that can't be shared among many people? Ask the workers themselves who they think are a good fit for the position. Chances are they know that hell of a lot better than a manager will. And it is abundantly clear how to get a promotion: make a good impression on your team. Cooperate, teach, help out, be nice, do good work. And these are -- by a weird coincidence -- just the things you want to accomplish.
- eadmund 7y ago
  I mostly agree with you, but: > And it is abundantly clear how to get a promotion: make a good impression on your team. Cooperate, teach, help out, be nice, do good work. … it strikes me that the problem is that the way to get a promotion would be to seem to cooperate, teach, help out, be nice & do good work, not necessarily actually do those things — one might end up with promotees who simply know how to game the system. It might still be preferable to the current state of affairs, which as you note hardly works.
GoodJokes 7y ago
Let me help you all save a lot of time. Authoritarian severely hierarchical organizations are inhumane and inefficient. Workplace democracy is the way to go.

177 points by edward 7y ago

TurkishPoptart 7y ago

Sadly, this article is paywalled for me.

emilburzo 7y ago
https://outline.com/JhCCJd https://outline.com/JhCCJd

Jaygles 7y ago

I get that its immeasurably frustrating for a person to be distilled into performance metrics, but how else can we measure one's impact? Using a trust based method where a supervisor leans on their personal biases to determine an employee's value is fraught with issues as well. A large enough corporation is a machine and we can't base decisions off of how we "feel" the machine's parts are working. Hard numbers will win over time.

jdc 7y ago
I think it really comes down to how well-designed the performance model is. And averaging a handful of survey answers sure as hell doesn't cut it.
Nasrudith 7y ago
The alternative is management who knows the domain well enough to understand those they supervise. That is expensive and hard in itself to ensure reliably. It is a matter of the push and pull of scale advantages at varied sizes essentially which is essentially universal but varies by domain.
lazyjeff 7y ago
Professors have been teaching and getting student course evaluations since forever. At institutions and roles where the admin knows little about the subject, they rely more on the course evals. At institutions with more engaged faculty and with more serious teaching, course evals are less prominent and defer to other forms of assessment (faculty observations, teaching statements, etc.).
- jld 7y ago
  Historically professors have tenure (admittedly less so now, I believe), so how they score on evaluations has no real bearing on them keeping their jobs. I had a professor who was a terrible lecturer, and he knew his student evaluations were terrible every time. As he handed out the evaluations he would tell us that he did not read them, and did not care what we wrote. I graduated twenty years ago, but just checked the department's website. He's still employed!
  - lazyjeff 7y ago
    Fair enough, but even the tenured professors were untenured at some point before so they did matter and were not problematic enough to deny tenure. And tenure doesn't really apply the same for adjuncts or some lecturers, who are perhaps even more dependent on student evals. And of course it depends on the seriousness the university takes teaching, as there are R1s that are more research oriented and care less about teaching, but they account for 1% of all universities.
    - virtuous_signal 7y ago
      Even as a grad student, my teaching evaluations couldn't negatively affect me. The only people it could have real, negative consequences for were instructors, who work on contracts. As an example, my department head (perhaps unintentionally) sent everyone an email saying any scores of less than 3.73 on a 1-7 scale were grounds for termination of that instructor's contract. I did not envy them at all, basically being highly educated customer service workers.
- hogFeast 7y ago
  Every large-scale student evaluation I have ever seen has been massively uninformative because students on different courses usually have drastically different expectations. At my university, the law school was top-notch (at least, it is the best in the country) and it got the lowest score by far every year. Why? High expectations, huge number of students competing hard. The best school? Theology. Why? Low expectations, not many students in a far less competitive environment (And I studied in both and in probably four or five faculties across the school...Law was the best, Theology the worst...not even close, the former had people from Law firms, you got full feedback on everything...the latter, you usually got no feedback on any work). This is really the point...now, you can do some kind of grouped mean model or you can compare over time...but what is the actual point? The reason they do this (or did this at my uni) was to compare departments/courses...but everyone knew it was bullshit...people lost jobs (I actually met the Dean who masterminded this, he came from the private sector, taught a minor course in the business school, and was utterly clueless).
vraivroo 7y ago
Yes, you've described why large corporations are hated. It's quite obvious that a handful of numbers are incapable of measuring a person's true impact, yet by god, they insist on trying.
- WalterBright 7y ago
  To be fair, metrics are often insisted upon by unions in order to not allow people to be fired based on the company's feelings about them. They're also insisted upon by courts to adjudicate wrongful termination cases as well as discrimination cases.
- tcbawo 7y ago
  All businesses want good employees, but what they want most are employees that can be replaced if necessary. They hire people to accomplish tasks and add value. I have felt underappreciated in corporate roles before, but then I realized that I was attaching too much of my identity to my job and in something where I had no real equity. Since then, I am much better at separating work from personal life, and I am happier putting my personal interests above the company's.
  - VRay 7y ago
    Yeah, I was a lot less upset about some of the crappy experiences I'd had working for evil giant companies after I read "The E-Myth Revisited". My elite 10 person team was doing a great job on Device A, while Device B had a team of 200 semi-skilled foreign contractors just barely keeping things on track. Then, somehow, the company decided to reorg my super-skilled team and promoted the leader of the contractor team. From 3-4 levels up in the company, the Device A team and the Device B team were IDENTICAL. They were both delivering things on time and under budget, even though team A was costing the company 1/5th as much, they were both fine. These products were generating hundreds of millions of dollars or maybe even billions in revenue per year, so the difference $5 million/year and $25 million was irrelevant. If I'd realized this sooner, I wouldn't have put so much effort into doing a great job at the expense of my own interests, and it would've been just fine
- 7y ago
hhs 7y ago
> I get that its immeasurably frustrating for a person to be distilled into performance metrics, but how else can we measure one's impact? I think it may also be useful to talk about the reliability and validity of all the things that are measured. Also, how useful is this stuff among the different fields? The article talks about gig workers and manufacturers. Would be interesting if these metrics were applied rigorously to other fields like politicians, talk show pundits, and forecasters.
- mjfl 7y ago
  This was the basic premise behind the book "Expert political judgement" by Phil Tetlock.
  - hhs 7y ago
    Thanks for the reference.
    - jacques_chester 7y ago
      It's an outstanding read. His followup Superforecasters was co-authored with a journalist for a more popular audience, but still fascinating.
- onceUponADime 7y ago
  No, no, this rigorious ratings are for lower class endavours. Higher classes get a title, which is a one off rubber stamp of quality and then will never again be judged on performance standing alone. Dr. Senior. Dipl. Ing. Except by there peers, who will lower a silk of omerta over "internal" affairs like a lack of quality in work. Of course, between these professions, there are meta-power struggles, which will play out, in a court- stylized fashion, a eternal ritualized war between castes like "management" and "technical execution". Metrics, as demanded by one caste, are worthless, as they rely on a working interface provided by the oppossition or in depth knowledge, making a allegiance two too factions necessary. Thus all that remains of them is a ritualized insistence, to compensate for the creeping feeling of a lack of controll. For those who need them, can not get them and be certain they are accurate. Making metrics, actually - a flag of truce.
ceejayoz 7y ago
> how else can we measure one's impact? This is a case of https://en.wikipedia.org/wiki/Politician%27s_syllogism https://en.wikipedia.org/wiki/Politician%27s_syllogism. We must do something This is something Therefore, we must do this.
- jschwartzi 7y ago
  Not to be confused with "Do something, even if it's wrong" which is how you break out of analysis paralysis.
- dredmorbius 7y ago
  A fave. See also: Don't just do something, stand there! https://quoteinvestigator.com/2014/03/22/stand-there/ https://quoteinvestigator.com/2014/03/22/stand-there/
grawprog 7y ago
>A large enough corporation is a machine A machine in which the parts of are made up of humans with 'feelings'. Distilling human beings down to cogs in a machine is bad enough when it comes from executives. This belief that corporations are some kind of non-human machine like entity is part of why the world's so fucked today. Maybe the world would be a better place if corporations weren't run as inhuman machines with no care or regard to their employees or the world in which they operate.
- jiggliemon 7y ago
  > This belief that corporations are some kind of non-human machine like entity is part of why the world's so fucked today. What point in history are you arguing was less fucked than it is today?
  - lstamour 7y ago
    Before corporations were people, in the US at least? http://reclaimdemocracy.org/corporate-accountability-history-corporations-us/ http://reclaimdemocracy.org/corporate-accountability-history... Not at all going to claim it as ideal, but it's a starting point for considering other possible alternatives to today's giant, faceless corporations where nobody within the corporation is easily held accountable, etc.
3pt14159 7y ago
I think the core problem is having one universal metric. Take Uber or Lyft as an example. Some people like silence, some like conversation, some like music. If the ML models were better or if it were based on a web of trust we wouldn't need to have a score that treats a 4.5 as universally better than a 4.6 or similar.
- deleted 7y ago
  [deleted]
hogFeast 7y ago
You should have more than one supervisor (and solicit information from peers). But basically...yes...using judgement, learning how to assess skills...that is all part of management. You cannot turn a bad manager into a good one with information. I have seen no evidence that "hard numbers" work (they aren't "hard"...there is usually almost no justification for a particular metric). And it feeds on innumerable human biases (inability to confront uncertainty, using "hard" information to justify an emotional conclusion, creating meaningless targets, etc.). In my experience, the practical issue is two-fold: one, people are just going to do what they do anyway but they ignore warnings signs/are more unflexible and two, you optimise for data that is obvious, collectable, and (usually) completely irrelevant. This is a trend, it will pass as all these trends do. It started off as quite a good idea (businesses were often failing to set targets or quantify success/failure) and has just got totally out of control.
moksly 7y ago
I work in the public sector where the pendulum is always swinging between trust and control. It's partly because we're not entirely driven by MBA's and our success criteria isn't earning shareholders value, but it's mainly because we've been around for so long because we can't go bankrupt when we fail. There are advantages and disadvantage to both approaches. In a system of trust the key benefits are better products, happier employees who have less sick days and stick around longer, more creativity and new solutions that are appreciated by the community. The downside is that some employees, managers or contractors are going to take advantage of the trust in such a terrible way that you'll have no option but to swing towards control once the media finds out. A good example from my country is a decade back, when we had a lot of trust in our public sector and an elderly care facility was saving money by putting dirty diapers back on the elderly if the urine in the diapers didn't weigh enough. Crazy, right? But that's the stuff that happens with trust, you'll also see managers contracting out procurement contracts to companies they own themselves and other stuff like that. On the opposite side you have the MBA based approach where everything is streamlined, measured and conformed. This is where you can build two hospitals for the price of 1,2 because the plan for them are exactly the same. It's where you LEAN every single business process up, and harvest the benefits. It's efficient, clean, and it slowly exhumes your business of value. Because the MBA approach very rarely manages to create something new, it's just really great at maximizing your current system. Eventually a neighboring city is going to do something wild, like figure out how to completely wipe out dyslexia by trying some crazy new method, because they trusted some hippie teacher with a plan. This is where you're forced to swing back toward trust and value tradeskills (not sure if this is the right word in English) because your politicians can't survive another period of having the highest dyslexia numbers in the country and nothing in the MBA playbook can help them. This is a little simplified of course, and I'm certain things are different in the private sector, but I dare say, that if your organisation sticks around for long enough, it's bound to see the pendulum swing between those two outlines.
- JamesBarney 7y ago
  I think the bigger issue with the MBA approach is it's always possible to trade a little bit of long term hard to measure value for short term profits.
taurath 7y ago
Hard numbers have been winning, and thats been the problem. Any time you have a set of numbers to describe a thing, you're creating an imperfect model of that thing. When you only consider that imperfect model, the things moves in the direction of fulfilling that model. You can't say you care about employee well-being, unless you have a metric, or several metrics, it won't ever be done.
Consultant32452 7y ago
I'm a bit on the spectrum and I absolutely love having my performance distilled down to a set of metrics. It clearly communicates to me which types of actions are valued in a way that I will often miss from normal social interactions due to my blindness to social queues. The way I perceive work is very helpful in this regard. To me, the purpose of work is to maximize my income/lifestyle while not dipping below a lower bound threshold of misery. I don't seek personal or emotional fulfillment from work at all. I did that when I was younger and, for me, it was a bad way to approach life. This way I'm much happier and my focus/attention is more on the kinds of things I hope will wind up in my obit: family, relationships, charity, etc. I wonder if sales people ever write articles about how they're at the "mercy" of their sales metrics. The idea of it seems hilarious.
- VRay 7y ago
  It's all well and good until you get judged by a bunch of completely borked-up, unfair metrics. The computer sets you to running halfway across a gigantic warehouse again and again, then measures your "picks per hour" vs somebody who consistently ends up in a tiny corner of the warehouse. Your manager agrees that it's not a problem and tells you not to worry, then a few months down the line some algorithm decides to give your lucky coworker a big raise and you don't get a boost at all. Thankfully I haven't had to deal with any metrics like this for more than a couple of months at a time.. Since engineers' specialty is to optimize ratings, it's an obvious-enough waste of time to apply them that they're usually short-lived.
CKN23-ARIN 7y ago
> how else can we measure one's impact? Goodhart's law suggests that this is impossible.
closeparen 7y ago
Yet people who passed laboratory science classes in college, people who should really know better, choose to govern by the metrics they "feel" are causally related to outcomes. Maybe you really do have a formula that outperforms a panel of subject matter experts. It does happen sometimes. Not for long, because the experts start using it. But the bar for making this kind of claim is, at minimum, peer-reviewed experimental results that reproduce. Human experience is a rich dataset. Intuition is a sophisticated machine. It blows my mind that people will so easily dismiss all of that in favor of a few bits of information and some 3rd grade arithmetic they just made up. It's not scientifically rigorous just because there's a table and a graph. Come on!

endtime 7y ago

I'd rather be at the mercy of _relatively_ objective metrics than at the mercy of whoever my boss happens to be. (I happen to like my current management chain, but people come and go...)

decebalus1 7y ago
Meh.. you're still at the mercy of whoever your boss happens to be. They just need to put a little effort into framing the metrics the right way or to make sure you're getting work that doesn't touch on the right metrics. The idea that performance metrics are fair is just corporate propaganda designed to make you have the incentive to improve them, thus being easier to control and replace.
- allset_ 7y ago
  >or to make sure you're getting work that doesn't touch on the right metrics. What? The metrics should be distilled from the overall company/org/team goals and mission and if you're working on things that don't align with those of course you're not going to have a good performance review.
  - gravitas 7y ago
    Think more subtly - if you need to meet a metric/goal of 50% of your time on phone calls and 50% on tickets, a manager who wants to tank your metrics will game the system to feed you less than needed to make your metric on one or the other. Just because a metric exists does not mean it's applied fairly or evenly to all employees.
- endtime 7y ago
  I said "relatively". Of course having a bad relationship with my boss would be bad for my career, no question.
nitwit005 7y ago
More generally, I'd prefer being judged by a large number of people rather than one or two managers who may or may not have any idea what's going on. I left my previous team partly because we went through three managers in less than a year, and I rarely saw them. I assume the performance feedback for the team was effectively nonsense.

makotech222 7y ago

This is the unstoppable march of capitalism, turning even its workers into commodities; predicted quite accurately by Marx in the 1800s. Half of our workforce will be able to be completely represented by some number. No different than a bushel of apples.

deleted 7y ago

[deleted]

CiPHPerCoder 7y ago

...does the "0.6 of a worker" bit remind anyone else of the "Three-Fifths compromise" from American History class? https://en.m.wikipedia.org/wiki/Three-Fifths_Compromise https://en.m.wikipedia.org/wiki/Three-Fifths_Compromise Not sure if an intentional dog whistle

WalterBright 7y ago

There's nothing new about this. My first job back in 1979 came with ratings used to determine raises. Corporations are rated, as well, on all sorts of numerical metrics.

greendestiny_re 7y ago

Hacker News should revel in this kind of headline; ratings are made by algorithms that can be gamed and exploited.

maxk42 7y ago

We've always been at the mercy of ratings. Now instead of being rated by a single person, we are rated by hundreds.

virtuous_signal 7y ago

One good thing to come out of certain rating systems, like Uber and Airbnb, is that the users get rated too. If someone is a terrible customer, then they can probably make things a lot worse for the driver or host than vice versa, and without these scores, drivers might not be able to warn other drivers about them. I would hope that with other modern customer service jobs, that reverse-ratings are always employed. It is a measurable way to improve the jobs of customer service workers.

aprdm 7y ago
I think there’s a bit of an imbalance as customers can easily create new accounts
- rock_hard 7y ago
  Not really...most of these services require to register with a phone number creating a pretty high barrier to create additional accounts for the common folk
  - xnyan 7y ago
    If you wanted to rent an Airbnb and you couldn’t because your phone number was associated with a negatively reviewed account, anyone who had the ability to get a phone number has the ability to get a second. It may be a minor inconvenience but not a particularly costly one. If I was a renter and wanted to disassociate from bad reviews, I imagine the easiest way to do it would be to transfer overship of the property to a holding company or similar. This is far harder to do and there’s far more cost.
  - gruez 7y ago
    >most of these services require to register with a phone number creating a pretty high barrier to create additional accounts for the common folk getting a prepaid sim is a "high barrier"?
    - jaxn 7y ago

hechang1997 7y ago

We should start a rating boycott where we rate highest for all ratings.

aphextron 7y ago
> We should start a rating boycott where we rate highest for all ratings. This company I've worked at for about a year now is the first place I've been exposed to worker ratings in my career, and this has been my strategy. I refuse to play the game, and give everyone a perfect score no matter what. It absolutely does not benefit us workers in any way to give someone less than perfect marks across the board. It is literally nothing but a tool used by management to provide justification when it comes time to downsize. Furthermore I've noticed that it tends to breed a culture of distrust and backstabbing among colleagues, who may use the opportunity as retribution for personal slights. It's an awful policy.

0xff00ffee 7y ago

A) Paywall. B) Get a different job.

ChainOfFools 7y ago

Les Measurables

jariel 7y ago

They need to do controlled trials whereby they try their best to truly and objectively measure a sample and then compare those to real ratings. I suggest that for most things, customer ratings are completely irrelevant. They might want to ask specific, relevant questions, such as: "Was the car/flat generally clean and tidy" "Were there any problems entering the facility" on the customer side "Did the customer generate problems with the authorities" "Did the customer disrupt your ability to drive" "Did the customer make repeated requests for out-of-bounds services even when they were informed such requests could not be fulfilled" "Did the customer party include more individuals than indicated on the reservation" etc. etc. Then they can glean specific bits of information and provide guidance. Otherwise, general reviews are pointless because they're muddy, set to different standards, emotional, unspecific etc.

kqr 7y ago

I seem to be the first to raise the Deming point of view again, but I'm glad to, so here goes: One of the big problems with trying to rate workers is that most workers have approximately the same rating. In other words, the differences in ratings are not statistically significant; they are just as well explained by pure chance. Building a system where you incentivise and promote on a random draw of cards is bad in and of itself, but doing it and then labeling it as being based on merit is actively hostile to anything you're trying to accomplish. You are building a culture of people stabbing each other in the back in desperation to win your lottery, and not a healthy environment of cooperation and focus on productivity. Yes, sometimes you have a worker who performs off the charts in either direction. That's a great opportunity! If they perform much better than someone else, put them in a position where they can coach and train others in how to do things that well. If they are much worse, try training them or otherwise find them a different job they can do better. Other than that special case, the only way to improve the performance of the system is to give the workers better tools, better information, better understanding, more cooperation, better ability to do their job. The manager is the only one standing in the way of the workers doing a better job. But then how do you assign bonuses and raises when everyone performs approximately the same? Split them across everyone. That simple. Think in terms of incentivising the team, not the individual. And things like promotions that can't be shared among many people? Ask the workers themselves who they think are a good fit for the position. Chances are they know that hell of a lot better than a manager will. And it is abundantly clear how to get a promotion: make a good impression on your team. Cooperate, teach, help out, be nice, do good work. And these are -- by a weird coincidence -- just the things you want to accomplish.

eadmund 7y ago
I mostly agree with you, but: > And it is abundantly clear how to get a promotion: make a good impression on your team. Cooperate, teach, help out, be nice, do good work. … it strikes me that the problem is that the way to get a promotion would be to seem to cooperate, teach, help out, be nice & do good work, not necessarily actually do those things — one might end up with promotees who simply know how to game the system. It might still be preferable to the current state of affairs, which as you note hardly works.

GoodJokes 7y ago

Let me help you all save a lot of time. Authoritarian severely hierarchical organizations are inhumane and inefficient. Workplace democracy is the way to go.