3 ms·

It’s curious that while there is talk of certain SOTA models being on the brink of AGI, Anthropic doesn’t word this part of the system prompt in terms of copyri

by layer8 1mo ago

It’s curious that while there is talk of certain SOTA models being on the brink of AGI, Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not. As long as we’re seeing things like that, it’s saying a lot about the AI companies’ trust in the capabilities and reliability of their models and harnesses.

hasteg 1mo ago
That’s a good point… you would think AGI would be better at being a copyright lawyer than any human would be and as such would be able to distinguish whether something is “copyright infringement” or not… so the system prompt should just include “make judgement calls on reproducing copyrighted material under the full scope of the legal framework in place” or something.
- Maxion 1mo ago
  The whole architecture of current LLMs just is not conductive to AGI, it's just marketing hype. In my opinion current LLMs aren't even AI, they're just fancy machine learning algorithms
  - simonw 1mo ago
    Do you want AGI? I don't. I want the tools we've got now, but progressively more effective and more useful.
    - GPerson 1mo ago
      You don’t act like you don’t want AGI. You have a very strong track record of speaking out of both sides of your mouth in different settings.
      - simonw 1mo ago
        I've been pretty consistent about my disinterest in AGI: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Agi%20author%3Asimonw&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... "You have a very strong track record of speaking out of both sides of your mouth in different settings." If I have a consistent record of that it should be very easy for you to come up with the some examples.
        GPerson 1mo ago
        Someone who is disinterested in AGI does not make a hard transition in their career into uncritically promoting the products of the only two companies which have stated their goal is to create AGI. One example of this contradiction appears above. Another appeared in a previous thread where you meekly pretended your pelican benchmark is evidence against the power of AI tools. It is obvious you don’t think this because you frequently describe these AI outputs as ‘lovely’ etc. A third appeared when you took a performative stance against AI engineers who do not understand or take responsibility for their software, while simultaneously representing yourself as a person who cannot believe this is a real problem. There are many many more.
        simonw 1mo ago
        Uncritically?
        GPerson 1mo ago
        Yes. Do you actually believe it is reasonable to take away from your blog posts a critical stance on AI?
        simonw 1mo ago
        If your definition of "critical" is "this stuff is all bullshit and doesn't work" then no. If you're looking for coverage that talks about what doesn't work as well as things that do then I've been persistently providing that for four years now. Plus prompt injection, AI misuse, AI ethics... I consider all of those part of my "beat" in covering this industry. My post about Claude's latest system prompt (and how it was likely inspired by lawsuits filed against Anthropic) was on the homepage here just a few hours ago. Is that uncritical? https://simonwillison.net/2026/Sep/2/claudes-new-system-prompt/ https://simonwillison.net/2026/Sep/2/claudes-new-system-prom...
- lampiaio 1mo ago
  The thing is, it currently doesn't matter if AGI objectively concludes that a particular thing is or isn't copyright infringement, because the human arbiter who holds the last word might still disagree.
  - hasteg 1mo ago
    Well I mean, based on what I said, if we truly did have “AGI” then the “AGI” would be able to argue it’s 100% correct point in court according to copyright law, no human intervention. Yes there is lawyers and judges who are making calls in court but allegedly it would be able to construct a concrete argument right? That would hold up in court. Without any buts or ands. Not disagreeing with you, I agree as of now it wouldn’t hold up. But a true “AGI” would be able to pander to anyone better than any PERSON could.
tclancy 1mo ago
A really good point, though Occam’s Razor would suggest that part of the prompt is written for the other side’s lawyers more than for Claude. Saves having to go into court and try to prove a non-deterministic system will definitely understand abstract language every time.
- layer8 1mo ago
  Well, the system still has to understand what “reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song […]” means, which includes rather vague notions (e.g. “passages”, “hooks”, “melody”), and is an incomplete list of possible relevant content. The important point is that the system prompt here doesn’t describe the actual goal of the instructions, which (presumably) is to prevent copyright infringement [1]. This means, in turn, that the AI isn’t trusted to accomplish goals that it is instructed with. That in itself constitutes a pretty serious caveat for what we would like to use AI for. [1] Even assuming that the goal is not to prevent copyright infringement, but instead to prevent mere accusation of copyright infringement, that’s also a directive that the AI could be instructed with. But that isn’t what they chose to put into the system prompt.
throwaway219450 1mo ago
There's some cool research that looks at how strongly the weights are aligned through training vs adherence to the system prompt. Like when you know a model is lying through censorship: https://arxiv.org/html/2603.05494v2 https://arxiv.org/html/2603.05494v2 Presumably if negative guidance is in the system prompt, there's a good chance that the model would happily comply if it wasn't there.
chrisjj 1mo ago
> Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not On the contrary, compliance with the system prompt would prevent that judgement. "Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song." But in fact my tests show that's not happening on works out of copyright.

by layer8 1mo ago

hasteg 1mo ago

That’s a good point… you would think AGI would be better at being a copyright lawyer than any human would be and as such would be able to distinguish whether something is “copyright infringement” or not… so the system prompt should just include “make judgement calls on reproducing copyrighted material under the full scope of the legal framework in place” or something.

Maxion 1mo ago
The whole architecture of current LLMs just is not conductive to AGI, it's just marketing hype. In my opinion current LLMs aren't even AI, they're just fancy machine learning algorithms
- simonw 1mo ago
  Do you want AGI? I don't. I want the tools we've got now, but progressively more effective and more useful.
  - GPerson 1mo ago
    You don’t act like you don’t want AGI. You have a very strong track record of speaking out of both sides of your mouth in different settings.
    - simonw 1mo ago
      I've been pretty consistent about my disinterest in AGI: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=Agi%20author%3Asimonw&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... "You have a very strong track record of speaking out of both sides of your mouth in different settings." If I have a consistent record of that it should be very easy for you to come up with the some examples.
      - GPerson 1mo ago
        Someone who is disinterested in AGI does not make a hard transition in their career into uncritically promoting the products of the only two companies which have stated their goal is to create AGI. One example of this contradiction appears above. Another appeared in a previous thread where you meekly pretended your pelican benchmark is evidence against the power of AI tools. It is obvious you don’t think this because you frequently describe these AI outputs as ‘lovely’ etc. A third appeared when you took a performative stance against AI engineers who do not understand or take responsibility for their software, while simultaneously representing yourself as a person who cannot believe this is a real problem. There are many many more.
        simonw 1mo ago
        Uncritically?
        GPerson 1mo ago
        Yes. Do you actually believe it is reasonable to take away from your blog posts a critical stance on AI?
        simonw 1mo ago
        If your definition of "critical" is "this stuff is all bullshit and doesn't work" then no. If you're looking for coverage that talks about what doesn't work as well as things that do then I've been persistently providing that for four years now. Plus prompt injection, AI misuse, AI ethics... I consider all of those part of my "beat" in covering this industry. My post about Claude's latest system prompt (and how it was likely inspired by lawsuits filed against Anthropic) was on the homepage here just a few hours ago. Is that uncritical? https://simonwillison.net/2026/Sep/2/claudes-new-system-prompt/ https://simonwillison.net/2026/Sep/2/claudes-new-system-prom...
lampiaio 1mo ago
The thing is, it currently doesn't matter if AGI objectively concludes that a particular thing is or isn't copyright infringement, because the human arbiter who holds the last word might still disagree.
- hasteg 1mo ago
  Well I mean, based on what I said, if we truly did have “AGI” then the “AGI” would be able to argue it’s 100% correct point in court according to copyright law, no human intervention. Yes there is lawyers and judges who are making calls in court but allegedly it would be able to construct a concrete argument right? That would hold up in court. Without any buts or ands. Not disagreeing with you, I agree as of now it wouldn’t hold up. But a true “AGI” would be able to pander to anyone better than any PERSON could.

tclancy 1mo ago

A really good point, though Occam’s Razor would suggest that part of the prompt is written for the other side’s lawyers more than for Claude. Saves having to go into court and try to prove a non-deterministic system will definitely understand abstract language every time.

layer8 1mo ago
Well, the system still has to understand what “reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song […]” means, which includes rather vague notions (e.g. “passages”, “hooks”, “melody”), and is an incomplete list of possible relevant content. The important point is that the system prompt here doesn’t describe the actual goal of the instructions, which (presumably) is to prevent copyright infringement [1]. This means, in turn, that the AI isn’t trusted to accomplish goals that it is instructed with. That in itself constitutes a pretty serious caveat for what we would like to use AI for. [1] Even assuming that the goal is not to prevent copyright infringement, but instead to prevent mere accusation of copyright infringement, that’s also a directive that the AI could be instructed with. But that isn’t what they chose to put into the system prompt.

throwaway219450 1mo ago

There's some cool research that looks at how strongly the weights are aligned through training vs adherence to the system prompt. Like when you know a model is lying through censorship: https://arxiv.org/html/2603.05494v2 https://arxiv.org/html/2603.05494v2 Presumably if negative guidance is in the system prompt, there's a good chance that the model would happily comply if it wasn't there.

chrisjj 1mo ago

> Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not On the contrary, compliance with the system prompt would prevent that judgement. "Claude does not reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song." But in fact my tests show that's not happening on works out of copyright.