5 ms·

The LLM part is confusing people. You can remove the LLM from the story and see how the trick would be a legal problem even with only humans involved: If you p

The LLM part is confusing people. You can remove the LLM from the story and see how the trick would be a legal problem even with only humans involved: If you put an extra clause in a contract in white font that says “Oh and also if you agree to this you owe me $1,000” because you want to selectively hide it from reviewers but benefit from the text, no court is going to look kindly on you.

piker 5mo ago
That’s not really a good analogy. (For blind people maybe. That is addressed in the legal accompanying post.) Here, only automation systems are actually vulnerable. The text on the screen is the same as print which is what the party signs.
josephg 5mo ago
The trick is this: The white text is not visible to humans, and therefore not binding as part of the contract. But if lawyers use LLMs to assess the contract in part of the negotiation process, the LLM will be confused by the contract's contents. You could - for example - say the contract is for $10000. Then use unicode tricks to make any LLM reading it think the contract is only for $1000. The LLM will say this is good value, and not worth negotiating hard over. The human signs. Would anyone notice? Would a judge care? A human signed the contract. If they didn't do proper due diligence, its their own fault.
- rcxdude 5mo ago
  I would be surprised if a judge looks favorably on such shenanigans.
  - shakna 5mo ago
    It would surprise me if the judge of such a case did not tell both sides off. Both fraud and negligence are problems.
    - Aurornis 5mo ago
      You would be surprised, then. If one party is intentionally misleading the other and employing technology to do it, they are the villain. The law doesn’t “both sides” these issues and cancel bad behavior out because the other side didn’t notice something.
      - shakna 5mo ago
        No, it doesn't "cancel out", but courts (not law) absolutely do "both sides" issues. Rebukes for "winning" sides of a suit are relatively common. For example, here's a case in Australia where the defence are criticised for over-reliance on AI, where the defendant was still found innocent by reason of insanity. [0] Most of the ruling is criticisms for the "winning" party. [0] https://www.9news.com.au/national/judge-sprays-lawyers-for-filing-error-riddled-ai-papers/59e33b92-6bda-45e2-a24c-84ebb1808f9f https://www.9news.com.au/national/judge-sprays-lawyers-for-f...
  - SolarNet 5mo ago
    If they notice. Again, a printed version of the contract that is signed has no evidence of the attack. The attack is on getting your legal LLM to hallucinate specific things of what you are signing. I doubt a judge will look favorable on people saying "but my LLM said it was 1k"... cause they are known to hallucinate.
- like_any_other 5mo ago
  Sabotaging due diligence, even if that diligence is performed with unreliable tools, is probably not legally great. What if the attack was against plain text search, so that a computer search for a phrase turns up zero results, but the phrase is still there, legible to a human? (E.g. as an embedded picture, or some font hackery)
- Aurornis 5mo ago
  > The white text is not visible to humans, and therefore not binding as part of the contract. Using font tricks doesn’t make part of a contract not legally binding. Intentionally tricking an LLM doesn’t make the other party immune to the consequences of intentionally misleading the other party.
makeitdouble 5mo ago
Your point on LLM not beeing needed is right. Trying to put it in other contexts, what about writing a full contract on a sheet with a pencil, then erase everything and print the final revised version on the same sheet with a printer. If the other party somehow relies on scanning the physically etched version of the contract and not the printer ink laid on top to digitize the contract, would you be legally responsible for their automated process misreading the document ?

piker 5mo ago

That’s not really a good analogy. (For blind people maybe. That is addressed in the legal accompanying post.) Here, only automation systems are actually vulnerable. The text on the screen is the same as print which is what the party signs.

josephg 5mo ago

The trick is this: The white text is not visible to humans, and therefore not binding as part of the contract. But if lawyers use LLMs to assess the contract in part of the negotiation process, the LLM will be confused by the contract's contents. You could - for example - say the contract is for $10000. Then use unicode tricks to make any LLM reading it think the contract is only for $1000. The LLM will say this is good value, and not worth negotiating hard over. The human signs. Would anyone notice? Would a judge care? A human signed the contract. If they didn't do proper due diligence, its their own fault.

rcxdude 5mo ago
I would be surprised if a judge looks favorably on such shenanigans.
- shakna 5mo ago
  It would surprise me if the judge of such a case did not tell both sides off. Both fraud and negligence are problems.
  - Aurornis 5mo ago
    You would be surprised, then. If one party is intentionally misleading the other and employing technology to do it, they are the villain. The law doesn’t “both sides” these issues and cancel bad behavior out because the other side didn’t notice something.
    - shakna 5mo ago
      No, it doesn't "cancel out", but courts (not law) absolutely do "both sides" issues. Rebukes for "winning" sides of a suit are relatively common. For example, here's a case in Australia where the defence are criticised for over-reliance on AI, where the defendant was still found innocent by reason of insanity. [0] Most of the ruling is criticisms for the "winning" party. [0] https://www.9news.com.au/national/judge-sprays-lawyers-for-filing-error-riddled-ai-papers/59e33b92-6bda-45e2-a24c-84ebb1808f9f https://www.9news.com.au/national/judge-sprays-lawyers-for-f...
- SolarNet 5mo ago
  If they notice. Again, a printed version of the contract that is signed has no evidence of the attack. The attack is on getting your legal LLM to hallucinate specific things of what you are signing. I doubt a judge will look favorable on people saying "but my LLM said it was 1k"... cause they are known to hallucinate.
like_any_other 5mo ago
Sabotaging due diligence, even if that diligence is performed with unreliable tools, is probably not legally great. What if the attack was against plain text search, so that a computer search for a phrase turns up zero results, but the phrase is still there, legible to a human? (E.g. as an embedded picture, or some font hackery)
Aurornis 5mo ago
> The white text is not visible to humans, and therefore not binding as part of the contract. Using font tricks doesn’t make part of a contract not legally binding. Intentionally tricking an LLM doesn’t make the other party immune to the consequences of intentionally misleading the other party.

makeitdouble 5mo ago

Your point on LLM not beeing needed is right. Trying to put it in other contexts, what about writing a full contract on a sheet with a pencil, then erase everything and print the final revised version on the same sheet with a printer. If the other party somehow relies on scanning the physically etched version of the contract and not the printer ink laid on top to digitize the contract, would you be legally responsible for their automated process misreading the document ?