4 ms·

> Before things like this become widespread, can we get some etiquette (and legal awareness)? I was watching CNBC interview last week with the founder of https

by cj 2y ago

> Before things like this become widespread, can we get some etiquette (and legal awareness)? I was watching CNBC interview last week with the founder of https://mercor.com/ https://mercor.com/ (backed by Benchmark, $250m valuation). The founder was pitching that their company would take every employee's employment and payroll history (even from prior roles) and use that to make AI recommendations to employers on things like compensation, employee retention, employee performance, etc. The majority of what the founder was describing would clearly be illegal if any human did it by hand. But somehow because a LLM is doing it, it becomes legal. Specific example: In most states it's illegal for a company to ask job candidates what their salary was in prior roles. But suddenly it's no longer illegal if a big company like ADP feeds all the data into a LLM and query against the LLM instead of the raw dataset. Copyright issues wasn't enough to regulate LLMs. But I suspect once we start seeing LLMs used in HR, performance reviews, pay raise decisions, hiring decisions, etc, people will start to care. [0] https://www.cnbc.com/video/2024/09/27/streamlining-hiring-with-ai-benchmark-backs-ai-start-up-mercor.html https://www.cnbc.com/video/2024/09/27/streamlining-hiring-wi...

roywiggins 2y ago
You don't need AI to get this data, Equifax already hoovers it all up and offers it as a service. https://theworknumber.com/solutions/products/income-employment-verification https://theworknumber.com/solutions/products/income-employme...
drdaeman 2y ago
> But somehow because a LLM is doing it, it becomes legal. IANAL, but I believe it does not. As it was famously said way back in the day, a computer can never be held accountable, therefore a computer must never make a management decision. There is always a human in the loop who makes the actual decision (even if that's just a formality), and if this decision is based on a flawed computer's recommendation, the flaws still apply. I think it was repeatedly proven that "company's computer says so" is not a legal defense.
- sterlind 2y ago
  aren't credit score checks literally this? credit scores are assigned to you based on feeding your personal data into some proprietary model. that score determines whether you get a loan, or a job, or a security clearance. there are policies that use hard cut-offs. how is that not exactly this?
  - consteval 2y ago
    It is exactly that (basically), and there's numerous ethical arguments around credit score. But credit score was in a somewhat unique position, because what predated it was obvious racism, sexism, and other types of discrimination. For both companies and consumers, it was a step up. Now, I'm not sure if that's the case. Today there's still many legal and moral qualms about using credit score for job applicants. It's illegal in many areas and highly scrutinized if a company does this.
  - drdaeman 2y ago
    Well, I haven't really kept track on this, but I believe some states (at least California and Illinois, I believe) prohibit use of credit scores for employment decisions, and I think there was some legislation banning this approved by the House but haven't passed Senate or something like that... So, yeah, you're right that it's an issue - and chances are we'll see a wider bans on this (governments are extremely slow).
kevin_thibedeau 2y ago
> take every employee's employment and payroll history (even from prior roles) Selling and purchasing employment history is thankfully banned in a growing number of states. Their business prospects in the US will eventually shrink to zero.
- cj 2y ago
  But is it illegal to buy a LLM model trained on such data? No, perfectly legal and acceptable (if we apply the same standards as we did for copyright).
  - drdaeman 2y ago
    I’m not a ML expert but I’m not sure a _language_ model makes sense here. A model like that… It’s basically going to be a ZIP code to salary coefficient mapping on steroids (with more parameters). The model by itself is probably (IANAL) legal if it can no longer produce any data points for individuals, but whenever using it for hiring purposes is legal or not certainly depends on inputs: e.g. feed it protected category (or a data that strongly correlates with one, e.g. name -> gender) and it most likely won’t fare well in court.

by cj 2y ago

roywiggins 2y ago

You don't need AI to get this data, Equifax already hoovers it all up and offers it as a service. https://theworknumber.com/solutions/products/income-employment-verification https://theworknumber.com/solutions/products/income-employme...

drdaeman 2y ago

> But somehow because a LLM is doing it, it becomes legal. IANAL, but I believe it does not. As it was famously said way back in the day, a computer can never be held accountable, therefore a computer must never make a management decision. There is always a human in the loop who makes the actual decision (even if that's just a formality), and if this decision is based on a flawed computer's recommendation, the flaws still apply. I think it was repeatedly proven that "company's computer says so" is not a legal defense.

sterlind 2y ago
aren't credit score checks literally this? credit scores are assigned to you based on feeding your personal data into some proprietary model. that score determines whether you get a loan, or a job, or a security clearance. there are policies that use hard cut-offs. how is that not exactly this?
- consteval 2y ago
  It is exactly that (basically), and there's numerous ethical arguments around credit score. But credit score was in a somewhat unique position, because what predated it was obvious racism, sexism, and other types of discrimination. For both companies and consumers, it was a step up. Now, I'm not sure if that's the case. Today there's still many legal and moral qualms about using credit score for job applicants. It's illegal in many areas and highly scrutinized if a company does this.
- drdaeman 2y ago
  Well, I haven't really kept track on this, but I believe some states (at least California and Illinois, I believe) prohibit use of credit scores for employment decisions, and I think there was some legislation banning this approved by the House but haven't passed Senate or something like that... So, yeah, you're right that it's an issue - and chances are we'll see a wider bans on this (governments are extremely slow).

kevin_thibedeau 2y ago

> take every employee's employment and payroll history (even from prior roles) Selling and purchasing employment history is thankfully banned in a growing number of states. Their business prospects in the US will eventually shrink to zero.

cj 2y ago
But is it illegal to buy a LLM model trained on such data? No, perfectly legal and acceptable (if we apply the same standards as we did for copyright).
- drdaeman 2y ago
  I’m not a ML expert but I’m not sure a _language_ model makes sense here. A model like that… It’s basically going to be a ZIP code to salary coefficient mapping on steroids (with more parameters). The model by itself is probably (IANAL) legal if it can no longer produce any data points for individuals, but whenever using it for hiring purposes is legal or not certainly depends on inputs: e.g. feed it protected category (or a data that strongly correlates with one, e.g. name -> gender) and it most likely won’t fare well in court.