9 ms·

Exploring portable ratings for gig workers

47 points by kawera 9y ago

crispyambulance 9y ago
Not sure how I feel about "portable" ratings. Sometimes people just need a fresh start and a chance to reinvent themsleves with a new gig in something totally new. If enough folks start carrying around their stars from one gig to another, employers are going to expect it. Gig workers who find themselves in a rut of poor performance for one reason or another just aren't going to be able to escape the old judgements of their performance for much longer periods of time. It's oppressive.
- ryanianian 9y ago
  It's a double-edged sword. I've hired some pros on Thumbtack and Taskrabbit that were industry veterans and had to struggle to re-earn their reputation when starting from fresh on gig sites. They probably really would have benefitted from being able to carry a reputation from other systems. Making reputation-follow "opt-in" makes a lot of sense. (Perhaps with egregiously negative reputations being weighted at least somewhat.)
  - TheAdamAndChe 9y ago
    Sure, that would make sense for the top 10% of performers. But most people aren't in the top 10%. They're already working in the "gig" economy, which is notorious for its instability and continued deleveraging of the worker. Wouldn't this just amplify those issues over time?
    - ryanianian 9y ago
      I don't see how it would amplify over time - in fact maybe the opposite. If you could opt-in to carry over, you build your reputation over time even as various gig marketplaces come and go. It encourages new marketplaces because workers don't have to worry about building up a good reputaion all over again on a new site, so they're probably gonna be more likely to try out new sites. It discourages scammers because customers know even workers that are new to the platform have built up a good reputation elsewhere, and they know that anything seriously bad will follow them. I see primary difficulty in how to communicate the "egregiously bad reviews will follow you" effect and how to determine which reviews are egregious. Plus it's like 100% probability of it being gamed with bad-actors somehow... (Personally I'd define the "egregiousness scale" to be a time-weighted average that's also weighted by the percentile of the reviewer's average rating - if you give all your gig workers 3 stars, then 3 stars is effectively 5 stars for you so workers see that benefit, and reviews further in the past count less against you both pro and con. But I'm not qualified to know if this would actually work out.)
      - lev99 9y ago
        It makes someone new to the economy have to work even harder to get a start. If there is a new gig platform and I have 0 rep, that's fine because everyone has 0 rep. If there is a new gig platform and I have 0 rep, and a lot of people brought their portable reputation with them, it means I have a disadvantage. It does provide more knowledge to consumers, and it does allow people to make different choices about who works for them.
        ryanianian 9y ago
        I like your notion of a level playing field, but I think you're basically saying we should ignore past experience on other marketpalces because it favors people with an established track-record? Isn't someone with a proven track-record exactly the kind of person we want to favor? Gig marketplaces come and go. If you spend 5 years working dozens of jobs a week to build up thousands of positive reviews only to have the marketplace go belly-up and now you have to start at zero, how do you think the gig economy can retain actual talent that expects to get paid well for their proven track-record? Or put it another way: I personally like the option of being able to pay more for people with a long history of positive reviews. I'm willing to pay much less to hire an untrusted worker. I don't think I'm alone. If everything starts at zero every few years, then all the wages reset to baseline every few years as well. Sure let people start at zero, but also let them get paid like an untrusted worker. And sure, let people bring over their experience. But let them get paid and charge like they have that experience.
        lev99 9y ago
        Describing who benefits from properties of one policy (non-transferable reputation) is not an inherit stance in favor of that policy. Every policy has winners and losers. I just felt like on this discussion thread there was a loser (people entering the gig economy) not mentioned.
        ryanianian 9y ago
        Good point.
- cloverich 9y ago
  I imagine the system could be given a bankruptcy outlet similar to credit ratings. There's definitely a balance here of reducing job-finding friction vs treating workers with poor work histories fairly. But I think the general argument that its outright oppressive is throwing the baby out with the bath water.
weego 9y ago
Hey it's The Circle but even more naive.
- dang 9y ago
  Please don't post snarky dismissals here. An informative comment can be critical while teaching the reader something, but this just puts down someone's work.
oldcynic 9y ago
There's a Black Mirror episode for this isn't there?
kristianc 9y ago
>> “Cab” drivers didn’t have visible habits around their ratings, weren’t checking them frequently and when we spoke about them, they told us that this wasn’t something they’d considered before or something they were particularly concerned about. They were confident in their skills and ability to find work outside of their platforms, and viewed ratings more as performance indicators for their platform owners — the main fear being a drop below 3.5 stars, where they might be dropped from the platform completely. This is a shorthand way of saying the drivers themselves aren't even checking their ratings, as long as they're above the performance threshold! This paragraph gives away who 'ratings portability' is really serving, and it is not the drivers.
- pavel_lishin 9y ago
  It's neither the drivers nor the customers, either. When I need a Lyft, I don't get a choice between three drivers with various star ratings - I enter my location, and someone accepts the ride. The rating is only good for the employer.
  - jjevanoorschot 9y ago
    The rating also benefits the customers. You don't get a choice between drivers, because only drivers with a high enough score will be allowed to continue on the platform. This might be harsh to the drivers, but it makes for a good customer experience. It's one of the reasons why the experience of using Uber/Lyft is so much better than the experience of using most traditional cabs.
    - Pamar 9y ago
      Unless Uber/Lyft decide that below-4-stars rated drivers can keep working for them... they just receive a smaller cut of the price paid by the customer.
kbart 9y ago
The fundamental problem of the rating systems is how to prevent one party or another (those who receive and those who write reviews) from abusing the system for their own advantage? The two obvious problems are: 1) fake reviews 2) blackmailing ("give me free stuff / discount / do what I say or I'll give you 1 star"). While the first problem is obvious, the latter is especially hard and I have yet to see it solved properly. The only place that comes to my mind that has somewhat mitigated it was early eBay that had ratings for both sellers and clients, so dishonest client would receive low ratings. But it also tended to discriminate new users as there's no way to distinguish between a honest, new user and just another single-use account for a scam. Furthermore, it only works on an established system, as you need to have many ratings to filter out noise, so it's a chicken/egg game ("nobody uses platform, because there's not enough users" / "there's not enough users, because nobody uses the platform"). I don't see how these problems could be solved without a 3rd party arbitration, but to do this you have to move back to square 1 (centralized service).
- ryanwaggoner 9y ago
  Even fake ratings are really hard to guard against in the individual case. You’re also forgetting sabotage by competitors.
- philipodonnell 9y ago
  Bi-directional reviewing from verified transactions seems to work well in dark markets on both of those problems, which are probably one of the marketplace structures with the least amount of in-built trust. You also see 'also find us on xxx market as yyy' which is an attempt at transferring reputation.
  - kbart 9y ago
    Yes, but as I've mentioned, you need a 3rd party arbitrage/platform on every transaction to achieve that which counters the "portable" idea.
jdlyga 9y ago
Definitely not. I saw the Black Mirror episode.
stevespang 9y ago
I use dozens of so called labor workers in Craig's Gigs, and I can tell you from experience most of them are an unreliable lot. Craig's seriously needs a rating system perhaps linked to phone no. at least, so that those who use these independent contractor laborers can have some way to screen them, regardless of the weaknesses. Most of them will agree to a payment and agree to show up at site, then just never show up at all. I have discovered what they do is solicit several offers with a YES, then choose the highest pay and never cancel the other offers. As a common courtesy, I only make arrangements with the workers I need, but lately I have had to counter the scammers by arranging with more workers than I need, to meet up at various street corners, all near the jobsite. After I fill my quota, I just go to the jobsite, then text all the others, sorry, don't need you. I wish there were a better way, but it's the abuse of their fellow workers that is hurting the entire labor pool.
lozenge 9y ago
> we felt like we had a sense that reputation portability was a real issue for some gig economy workers But it wasn't an issue for the "gig economy" platforms. They are happy to lock in workers by not exporting ratings. On top of that, accepting imported ratings would provide a boost to the competitors that export ratings. So, what's the business case? Pretty design, but I'm amazed somebody could write an article covering so many detailed questions and not even touch on this rather fundamental one. It reminds me a bit of the heady OpenID days where every piddling website was an OpenID IdP (identity provider), but next to no websites actually accepted OpenID logins.
frgtpsswrdlame 9y ago
This is so indicative of the difference between the US and China. So in China they're looking at a Social Credit System (which has been talked about here before [1][2]) which is a government program which would apply to all Chinese citizens and rate their trustworthiness. It's seen as authoritarian, providing no way to start over, etc. Here in America we've got this post and it's implication that we're going to institute a similar social credit system except it will come from corporations, not government and it won't apply to everyone, only gig workers. It's seen as an interesting technical problem and a way to increase market efficiency. A tale of two viewpoints. [1] https://news.ycombinator.com/item?id=13201926 https://news.ycombinator.com/item?id=13201926 [2] https://news.ycombinator.com/item?id=9445538 https://news.ycombinator.com/item?id=9445538
- anentropic 9y ago
  quite chilling either way
- dragonwriter 9y ago
  > Here in America we've got this post and ...criticism in this very thread they almost exactly mirrors the criticism you point to by Americans of the Chinese system, particularly on the “no way to start over”.
  - TheAdamAndChe 9y ago
    From the way I interpreted their post, that was their point. We like to complain about the bad system that China is implementing, yet our market system is doing the exact same thing for the same reasons as China.
- oldcynic 9y ago
  When I started writing this most comments are pretty negative about applying this to gig workers. In essence they're one and the same. China: Manage to upset, doesn't matter how, the party representative and you're doomed. West: s/party/company/ Neither approach brings truth, just verisimilitude. Every review or score out there has been gamed in some way either postively by friends and colleagues or negatively by your competitors.
ilaksh 9y ago
We need portable data for everyone and everything, but we also need to be careful to do it in a way that doesn't become oppressive and centralized or inflexible. So there should be some compatibility but not a single system or rating.
mr_spothawk 9y ago
Can we call it "day labor" from now on?
ejlangev 9y ago
Social credit imposed by corporate rather than state power isn't any less of a bad idea. Perhaps there are reasons why every decision a person has ever made shouldn't follow them forever such that they need to show it to their future employers. A better system would be to just force people to wear their overall positive rating percentage visibly on their clothes in large red print. We can call it... the scarlet number.
niftich 9y ago
From a technical standpoint, this is largely the same problem as the portability of other user data, with the added complication that the data must be signed by an identifiable originating party other than the user that the receiving party can correspond -- and establish a trust relationship with -- to verify the lack of tampering. Online services have been around for a long time and yet very few implement data portability. Often, it's part of their moat against competitors and ensures lock-in. Few incentives exist for incumbents to offer data export. From a social standpoint, this raises some serious issues. Distributed reputation systems exist informally, whose lack of universal applicability give them some protection against trivial malice. In the financial world, various credit scoring systems use trusted third parties to aggregate information from entities one has transacted with, and offer a service for other parties to obtain this information on the user's request. In some places government-issued strong authentication is used to establish identity, but in the US, weak identification and publicly available information is used as a challenge to the user to establish their identity. Both of them come with a host of issues that ought to be cause for concern. If no link to some "higher form" of identity is implied, then the problem becomes almost trivial: each user of 'Service A' can become a cross-referenced user on 'Service B' by proving control over their account in 'Service A'. But this too is vulnerable to credential theft, and new guises can be created at any time by just creating additional accounts in 'Service A'.
m0llusk 9y ago
The biggest problem is objective metrics. Did someone try to make contact when something came up? Were they on time? Did the work get done properly? Did the provider behave in a professional manner? All of the most important judgments are inherently subjective and biased. This then leads up the chain into the problem of rating the rating reputations of the raters.
walshemj 9y ago
Why would you need this for these "gig" economy jobs as they are all low/no skill jobs all that's required is that you have no alternate job and are desperate.
zackmorris 9y ago
Since these systems all attempt to extrapolate future performance based on past experience, they are almost by definition an affront to free will. The problem is that when something as important as someone's livelihood depends on factors outside their control (like someone else's subjective impression of their performance) then it opens the system up to corruption/gaming/extortion. It's probably valid to keep a log of someone's employee history. But it must be under that person's control and they must have avenues for disputing its contents or expunging it in its entirety (similar to a bankruptcy). If we go down this road, then it must be balanced by employer ratings such as payment followthrough.
JustSomeNobody 9y ago
How do you keep it from being gamed against women and minorities[0]? [0] For example.
delinka 9y ago
I was recently reading [1]. In Chapter 2, this kind of 'ratings sharing' is detrimental to your own employment. Didn't work so well at the last place? The software remembers and won't hire you for the new place. If that's where this is headed, it won't be good. http://marshallbrain.com/manna1.htm http://marshallbrain.com/manna1.htm

47 points by kawera 9y ago

Not sure how I feel about "portable" ratings. Sometimes people just need a fresh start and a chance to reinvent themsleves with a new gig in something totally new. If enough folks start carrying around their stars from one gig to another, employers are going to expect it. Gig workers who find themselves in a rut of poor performance for one reason or another just aren't going to be able to escape the old judgements of their performance for much longer periods of time. It's oppressive.

ryanianian 9y ago
It's a double-edged sword. I've hired some pros on Thumbtack and Taskrabbit that were industry veterans and had to struggle to re-earn their reputation when starting from fresh on gig sites. They probably really would have benefitted from being able to carry a reputation from other systems. Making reputation-follow "opt-in" makes a lot of sense. (Perhaps with egregiously negative reputations being weighted at least somewhat.)
- TheAdamAndChe 9y ago
  Sure, that would make sense for the top 10% of performers. But most people aren't in the top 10%. They're already working in the "gig" economy, which is notorious for its instability and continued deleveraging of the worker. Wouldn't this just amplify those issues over time?
  - ryanianian 9y ago
    I don't see how it would amplify over time - in fact maybe the opposite. If you could opt-in to carry over, you build your reputation over time even as various gig marketplaces come and go. It encourages new marketplaces because workers don't have to worry about building up a good reputaion all over again on a new site, so they're probably gonna be more likely to try out new sites. It discourages scammers because customers know even workers that are new to the platform have built up a good reputation elsewhere, and they know that anything seriously bad will follow them. I see primary difficulty in how to communicate the "egregiously bad reviews will follow you" effect and how to determine which reviews are egregious. Plus it's like 100% probability of it being gamed with bad-actors somehow... (Personally I'd define the "egregiousness scale" to be a time-weighted average that's also weighted by the percentile of the reviewer's average rating - if you give all your gig workers 3 stars, then 3 stars is effectively 5 stars for you so workers see that benefit, and reviews further in the past count less against you both pro and con. But I'm not qualified to know if this would actually work out.)
    - lev99 9y ago
      It makes someone new to the economy have to work even harder to get a start. If there is a new gig platform and I have 0 rep, that's fine because everyone has 0 rep. If there is a new gig platform and I have 0 rep, and a lot of people brought their portable reputation with them, it means I have a disadvantage. It does provide more knowledge to consumers, and it does allow people to make different choices about who works for them.
      - ryanianian 9y ago
        I like your notion of a level playing field, but I think you're basically saying we should ignore past experience on other marketpalces because it favors people with an established track-record? Isn't someone with a proven track-record exactly the kind of person we want to favor? Gig marketplaces come and go. If you spend 5 years working dozens of jobs a week to build up thousands of positive reviews only to have the marketplace go belly-up and now you have to start at zero, how do you think the gig economy can retain actual talent that expects to get paid well for their proven track-record? Or put it another way: I personally like the option of being able to pay more for people with a long history of positive reviews. I'm willing to pay much less to hire an untrusted worker. I don't think I'm alone. If everything starts at zero every few years, then all the wages reset to baseline every few years as well. Sure let people start at zero, but also let them get paid like an untrusted worker. And sure, let people bring over their experience. But let them get paid and charge like they have that experience.
        lev99 9y ago
        Describing who benefits from properties of one policy (non-transferable reputation) is not an inherit stance in favor of that policy. Every policy has winners and losers. I just felt like on this discussion thread there was a loser (people entering the gig economy) not mentioned.
        ryanianian 9y ago
        Good point.
cloverich 9y ago
I imagine the system could be given a bankruptcy outlet similar to credit ratings. There's definitely a balance here of reducing job-finding friction vs treating workers with poor work histories fairly. But I think the general argument that its outright oppressive is throwing the baby out with the bath water.

weego 9y ago

Hey it's The Circle but even more naive.

dang 9y ago
Please don't post snarky dismissals here. An informative comment can be critical while teaching the reader something, but this just puts down someone's work.

oldcynic 9y ago

There's a Black Mirror episode for this isn't there?

kristianc 9y ago

>> “Cab” drivers didn’t have visible habits around their ratings, weren’t checking them frequently and when we spoke about them, they told us that this wasn’t something they’d considered before or something they were particularly concerned about. They were confident in their skills and ability to find work outside of their platforms, and viewed ratings more as performance indicators for their platform owners — the main fear being a drop below 3.5 stars, where they might be dropped from the platform completely. This is a shorthand way of saying the drivers themselves aren't even checking their ratings, as long as they're above the performance threshold! This paragraph gives away who 'ratings portability' is really serving, and it is not the drivers.

pavel_lishin 9y ago
It's neither the drivers nor the customers, either. When I need a Lyft, I don't get a choice between three drivers with various star ratings - I enter my location, and someone accepts the ride. The rating is only good for the employer.
- jjevanoorschot 9y ago
  The rating also benefits the customers. You don't get a choice between drivers, because only drivers with a high enough score will be allowed to continue on the platform. This might be harsh to the drivers, but it makes for a good customer experience. It's one of the reasons why the experience of using Uber/Lyft is so much better than the experience of using most traditional cabs.
  - Pamar 9y ago
    Unless Uber/Lyft decide that below-4-stars rated drivers can keep working for them... they just receive a smaller cut of the price paid by the customer.

kbart 9y ago

The fundamental problem of the rating systems is how to prevent one party or another (those who receive and those who write reviews) from abusing the system for their own advantage? The two obvious problems are: 1) fake reviews 2) blackmailing ("give me free stuff / discount / do what I say or I'll give you 1 star"). While the first problem is obvious, the latter is especially hard and I have yet to see it solved properly. The only place that comes to my mind that has somewhat mitigated it was early eBay that had ratings for both sellers and clients, so dishonest client would receive low ratings. But it also tended to discriminate new users as there's no way to distinguish between a honest, new user and just another single-use account for a scam. Furthermore, it only works on an established system, as you need to have many ratings to filter out noise, so it's a chicken/egg game ("nobody uses platform, because there's not enough users" / "there's not enough users, because nobody uses the platform"). I don't see how these problems could be solved without a 3rd party arbitration, but to do this you have to move back to square 1 (centralized service).

ryanwaggoner 9y ago
Even fake ratings are really hard to guard against in the individual case. You’re also forgetting sabotage by competitors.
philipodonnell 9y ago
Bi-directional reviewing from verified transactions seems to work well in dark markets on both of those problems, which are probably one of the marketplace structures with the least amount of in-built trust. You also see 'also find us on xxx market as yyy' which is an attempt at transferring reputation.
- kbart 9y ago
  Yes, but as I've mentioned, you need a 3rd party arbitrage/platform on every transaction to achieve that which counters the "portable" idea.

jdlyga 9y ago

Definitely not. I saw the Black Mirror episode.

stevespang 9y ago

I use dozens of so called labor workers in Craig's Gigs, and I can tell you from experience most of them are an unreliable lot. Craig's seriously needs a rating system perhaps linked to phone no. at least, so that those who use these independent contractor laborers can have some way to screen them, regardless of the weaknesses. Most of them will agree to a payment and agree to show up at site, then just never show up at all. I have discovered what they do is solicit several offers with a YES, then choose the highest pay and never cancel the other offers. As a common courtesy, I only make arrangements with the workers I need, but lately I have had to counter the scammers by arranging with more workers than I need, to meet up at various street corners, all near the jobsite. After I fill my quota, I just go to the jobsite, then text all the others, sorry, don't need you. I wish there were a better way, but it's the abuse of their fellow workers that is hurting the entire labor pool.

lozenge 9y ago

> we felt like we had a sense that reputation portability was a real issue for some gig economy workers But it wasn't an issue for the "gig economy" platforms. They are happy to lock in workers by not exporting ratings. On top of that, accepting imported ratings would provide a boost to the competitors that export ratings. So, what's the business case? Pretty design, but I'm amazed somebody could write an article covering so many detailed questions and not even touch on this rather fundamental one. It reminds me a bit of the heady OpenID days where every piddling website was an OpenID IdP (identity provider), but next to no websites actually accepted OpenID logins.

frgtpsswrdlame 9y ago

This is so indicative of the difference between the US and China. So in China they're looking at a Social Credit System (which has been talked about here before [1][2]) which is a government program which would apply to all Chinese citizens and rate their trustworthiness. It's seen as authoritarian, providing no way to start over, etc. Here in America we've got this post and it's implication that we're going to institute a similar social credit system except it will come from corporations, not government and it won't apply to everyone, only gig workers. It's seen as an interesting technical problem and a way to increase market efficiency. A tale of two viewpoints. [1] https://news.ycombinator.com/item?id=13201926 https://news.ycombinator.com/item?id=13201926 [2] https://news.ycombinator.com/item?id=9445538 https://news.ycombinator.com/item?id=9445538

anentropic 9y ago
quite chilling either way
dragonwriter 9y ago
> Here in America we've got this post and ...criticism in this very thread they almost exactly mirrors the criticism you point to by Americans of the Chinese system, particularly on the “no way to start over”.
- TheAdamAndChe 9y ago
  From the way I interpreted their post, that was their point. We like to complain about the bad system that China is implementing, yet our market system is doing the exact same thing for the same reasons as China.
oldcynic 9y ago
When I started writing this most comments are pretty negative about applying this to gig workers. In essence they're one and the same. China: Manage to upset, doesn't matter how, the party representative and you're doomed. West: s/party/company/ Neither approach brings truth, just verisimilitude. Every review or score out there has been gamed in some way either postively by friends and colleagues or negatively by your competitors.

ilaksh 9y ago

We need portable data for everyone and everything, but we also need to be careful to do it in a way that doesn't become oppressive and centralized or inflexible. So there should be some compatibility but not a single system or rating.

mr_spothawk 9y ago

Can we call it "day labor" from now on?

ejlangev 9y ago

Social credit imposed by corporate rather than state power isn't any less of a bad idea. Perhaps there are reasons why every decision a person has ever made shouldn't follow them forever such that they need to show it to their future employers. A better system would be to just force people to wear their overall positive rating percentage visibly on their clothes in large red print. We can call it... the scarlet number.

niftich 9y ago

From a technical standpoint, this is largely the same problem as the portability of other user data, with the added complication that the data must be signed by an identifiable originating party other than the user that the receiving party can correspond -- and establish a trust relationship with -- to verify the lack of tampering. Online services have been around for a long time and yet very few implement data portability. Often, it's part of their moat against competitors and ensures lock-in. Few incentives exist for incumbents to offer data export. From a social standpoint, this raises some serious issues. Distributed reputation systems exist informally, whose lack of universal applicability give them some protection against trivial malice. In the financial world, various credit scoring systems use trusted third parties to aggregate information from entities one has transacted with, and offer a service for other parties to obtain this information on the user's request. In some places government-issued strong authentication is used to establish identity, but in the US, weak identification and publicly available information is used as a challenge to the user to establish their identity. Both of them come with a host of issues that ought to be cause for concern. If no link to some "higher form" of identity is implied, then the problem becomes almost trivial: each user of 'Service A' can become a cross-referenced user on 'Service B' by proving control over their account in 'Service A'. But this too is vulnerable to credential theft, and new guises can be created at any time by just creating additional accounts in 'Service A'.

m0llusk 9y ago

The biggest problem is objective metrics. Did someone try to make contact when something came up? Were they on time? Did the work get done properly? Did the provider behave in a professional manner? All of the most important judgments are inherently subjective and biased. This then leads up the chain into the problem of rating the rating reputations of the raters.

walshemj 9y ago

Why would you need this for these "gig" economy jobs as they are all low/no skill jobs all that's required is that you have no alternate job and are desperate.

zackmorris 9y ago

Since these systems all attempt to extrapolate future performance based on past experience, they are almost by definition an affront to free will. The problem is that when something as important as someone's livelihood depends on factors outside their control (like someone else's subjective impression of their performance) then it opens the system up to corruption/gaming/extortion. It's probably valid to keep a log of someone's employee history. But it must be under that person's control and they must have avenues for disputing its contents or expunging it in its entirety (similar to a bankruptcy). If we go down this road, then it must be balanced by employer ratings such as payment followthrough.

JustSomeNobody 9y ago

How do you keep it from being gamed against women and minorities[0]? [0] For example.

delinka 9y ago

I was recently reading [1]. In Chapter 2, this kind of 'ratings sharing' is detrimental to your own employment. Didn't work so well at the last place? The software remembers and won't hire you for the new place. If that's where this is headed, it won't be good. http://marshallbrain.com/manna1.htm http://marshallbrain.com/manna1.htm