4 ms·

This feels like the OSS community is giving up. LLMs are lossily-compressed models of code and other text (often mass-scraped despite explicit non-consent) whi

This feels like the OSS community is giving up. LLMs are lossily-compressed models of code and other text (often mass-scraped despite explicit non-consent) which has licenses almost always requiring attribution and very often other conditions. Just a few weeks ago a SOTA model was shown to reproduce non-trivial amounts of licensed code[0]. The idea of intelligence being emergent from compression is nothing new[1]. The trick here is giving up on completeness and accuracy in favor of a more probabilistic output which 1) reproduces patterns and interpolates between patterns of training data while not always being verbatim copies 2) serves as a heuristic when searching the solution-space which is further guided by deterministic tools such as compilers, linters, etc. - the models themselves quite often generate complete nonsense, including making up non-existent syntax in well-known mainstream languages such as C#. I strongly object to anthropomorphising text transformers (e.g. "Assisted-by"). It encourages magical thinking even among people who understand how the models operate, let alone the general public. Just like stealing fractional amounts of money[3] should not be legal, violating the licenses of the training data by reusing fractional amounts from each should not be legal either. [0]: https://news.ycombinator.com/item?id=47356000 https://news.ycombinator.com/item?id=47356000 [1]: http://prize.hutter1.net/ http://prize.hutter1.net/ [2]: https://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect [3]: https://skeptics.stackexchange.com/questions/14925/has-a-programmer-ever-embezzled-money-by-shaving-fractions-of-a-cent-from-many-b https://skeptics.stackexchange.com/questions/14925/has-a-pro...

tmp10423288442 6mo ago
On https://news.ycombinator.com/item?id=47356000 https://news.ycombinator.com/item?id=47356000, it looks like the user there was intentionally asking about the implementation of the Python chardet library before asking it to write code, right? Not surprising the AI would download the library to investigate it by default, or look for any installed copies of `chardet` on the local machine.
- martin-t 6mo ago
  The comment says "Opus 4.6 without tool use or web access"
KK7NIL 6mo ago
> I strongly object to anthropomorphising text transformers (e.g. "Assisted-by"). I don't think this is anthropomorphising, especially considering they also include non-LLM tools in that "Assisted-by" section. We're well past the Turing test now, whether these things are actually sentient or not is of no pragmatic importance if we can't distinguish their output from a sentient creature, especially when it comes to programming.
- martin-t 6mo ago
  Would you say "assisted by vim" or "assisted by gcc"? It should be either something like "(partially/completely) generated by" or if you want to include deterministic tools, then "Tools-used:". The Turing test is an interesting thought experiment but we've seen it's easy for LLMs to sound human-like or make authoritative and convincing statements despite being completely wrong or full of nonsense. The Turing test is not a measure of intelligence, at least not an artificial one. (Though I find it quite amusing to think that the point at which a person chooses to refer to LLMs as intelligence is somewhat indicative of his own intelligence level.) > whether these things are actually sentient or not is of no pragmatic importance if we can't distinguish their output from a sentient creature, especially when it comes to programming It absolutely makes a difference: you can't own a human but you can own an LLM (or a corporation which is IMO equally wrong as owning a human). Humans have needs which must be continually satisfied to remain alive. Humans also have a moral value (a positive one - at least for most of us) which dictates that being rendered unable to remain alive is wrong. Now, what happens if LLMs have the same legal standing as humans and are thus able to participate in the economy in the same manner?
  - zbentley 6mo ago
    If a linter insists on a weird line of code, I’m probably commenting that line as “recommended by whatever-linter”, yes.
    - martin-t 6mo ago
      I wouldn't but I can see why some people would. I can't point out where I draw the line clearly but here's one different I notice: A recommendation can be both a thing and an action. A piece of text is a recommendation and it does not matter how it was created. Assistance implies some parity in capabilities and cooperative work. Also it can pretty much only be an action, you cannot say "here is some assistance" and point to a thing.
ninjagoo 6mo ago
> Just like stealing fractional amounts of money[3] should not be legal, violating the licenses of the training data by reusing fractional amounts from each should not be legal either. I think you'll find that this is not settled in the courts, depending on how the data was obtained. If the data was obtained legally, say a purchased book, courts have been finding that using it for training is fair use (Bartz v. Anthropic, Kadrey v. Meta). Morally the case gets interesting. Historically, there was no such thing as copyright. The English 1710 Statute of Anne establishing copyright as a public law was titled 'for the Encouragement of Learning' and the US Constitution said 'Congress may secure exclusive rights to promote the progress of science and useful arts'; so essentially public benefits driven by the grant of private benefits. The Moral Bottomline: if you didn't have to eat, would you care about who copies your work as long as you get credited? The more the people that copy your work with attribution, the more famous you'll be. Now that's the currency of the future*. [1] You'll do it for the kudos. [2][3] *Post-Scarcity Future. [1] https://en.wikipedia.org/wiki/Post-scarcity [2] https://en.wikipedia.org/wiki/The_Quiet_War, et. al. [3] https://en.wikipedia.org/wiki/Accelerando
- martin-t 6mo ago
  > The Moral Bottomline: if you didn't have to eat, would you care about who copies your work as long as you get credited? Yes. I have 2 issues with "post-scarcity": - It often implicitly assumes humanity is one homogeneous group where this state applies to everyone. In reality, if post-scarcity is possible, some people will be lucky enough to have the means to live that lifestyle while others will still by dying of hunger, exposure and preventable diseases. All else being equal, I'd prefer being in the first group and my chance for that is being economically relevant. - It often ignores that some people are OK with having enough while others have a need to have more than others, no matter how much they already have. The second group is the largest cause of exploitation and suffering in the world. And the second group will continue existing in a post-scarcity world and will work hard to make scarcity a real thing again. --- Back to your question: I made the mistake of publishing most of my public code under GPL or AGPL. I regret is because even though my work has brought many people some joy and a bit of my work was perhaps even useful, it has also been used by people who actively enjoy hurting others, who have caused measurable harm and who will continue causing harm as long as they're able to - in a small part enabled by my code. Permissive licenses are socially agnostic - you can use the work and build on top of it no matter who you are and for what purpose. A(GPL) is weakly pro-social - you can use the work no matter what but you can only build on top of it if you give back - this produces some small but non-zero social pressure (enforced by violence through governments) which benefits those who prefer cooperation instead of competition. What I want is a strongly pro-social license - you can use or build on top of my work only if you fulfill criteria I specify such as being a net social good, not having committed any serious offenses, not taking actions to restrict other people's rights without a valid reason, etc. There have been attempts in this direction[0] but not very successful. In a world without LLMs, I'd be writing code using such a license but more clearly specified, even if I had to write my own. Yes, a layer would do a better job, that does not mean anything written by a non-lawyer is completely unenforceable. With LLMs, I have stopped writing public code at all because the way I see it, it just makes people much richer than me even richer at a much faster rate than I can ever achieve myself. Ir just makes inequality worse. And with inequality, exploitation and oppression tends to soon follow. [0]: https://json.org/license.html https://json.org/license.html
deleted 6mo ago
[deleted]
user34283 6mo ago
For [0], it was supposedly shown to do it when specifically prompted to do so. Despite agentic tools being used by millions of developers now, I am not aware of a single real case where accidental reproduction of copyrightable code has been an issue. Further, some model providers offer indemnity clauses. It seems like a non-issue to me, practically.
williamcotton 6mo ago
"Just a few weeks ago a SOTA model was shown to reproduce non-trivial amounts of licensed code[0]." That LLM response is describing a specific project with full attribution.
- martin-t 6mo ago
  And it proves the code is stored (in a compressed form) in the model.
  - williamcotton 6mo ago
    So what's the legal issue here? How does the chardet achieve this? Explain in detail, with shortened code excerpts from the library itself if helpful to the explanation. The prompt is explicitly requesting the source!
    - martin-t 6mo ago
      You're saying the same thing like it's relevant. What matters to me is that the source code is clearly stored encoded/compressed in the model. That to me means the license of the code should apply to the model. Now we just need somebody to ask it to decompress ("reproduce") AGPL code and its owners will have to release it under the AGPL as well. Yes, this is sarcasm but that's how it should work. These models cannot be created without ingesting vast amounts of code created by human labor. The proponents of this kind of stealing argue that the models "generalize" and that somehow makes it right and legal. And here we have counterexample to their generalization argument. Doesn't change the fact labor theft is still labor theft but it might make a difference to the letter of the law, hopefully. I say hopefully because I believe all people who contribute to technical progress should be compensated according to their amount of work and skill level. But of course many "temporarily embarrassed trillionaires" disagree. I don't understand why but I am happy to know that if the tides turn they will be perfectly A-OK with me taking their work without credit or compensation.

tmp10423288442 6mo ago

On https://news.ycombinator.com/item?id=47356000 https://news.ycombinator.com/item?id=47356000, it looks like the user there was intentionally asking about the implementation of the Python chardet library before asking it to write code, right? Not surprising the AI would download the library to investigate it by default, or look for any installed copies of `chardet` on the local machine.

martin-t 6mo ago
The comment says "Opus 4.6 without tool use or web access"

KK7NIL 6mo ago

> I strongly object to anthropomorphising text transformers (e.g. "Assisted-by"). I don't think this is anthropomorphising, especially considering they also include non-LLM tools in that "Assisted-by" section. We're well past the Turing test now, whether these things are actually sentient or not is of no pragmatic importance if we can't distinguish their output from a sentient creature, especially when it comes to programming.

martin-t 6mo ago
Would you say "assisted by vim" or "assisted by gcc"? It should be either something like "(partially/completely) generated by" or if you want to include deterministic tools, then "Tools-used:". The Turing test is an interesting thought experiment but we've seen it's easy for LLMs to sound human-like or make authoritative and convincing statements despite being completely wrong or full of nonsense. The Turing test is not a measure of intelligence, at least not an artificial one. (Though I find it quite amusing to think that the point at which a person chooses to refer to LLMs as intelligence is somewhat indicative of his own intelligence level.) > whether these things are actually sentient or not is of no pragmatic importance if we can't distinguish their output from a sentient creature, especially when it comes to programming It absolutely makes a difference: you can't own a human but you can own an LLM (or a corporation which is IMO equally wrong as owning a human). Humans have needs which must be continually satisfied to remain alive. Humans also have a moral value (a positive one - at least for most of us) which dictates that being rendered unable to remain alive is wrong. Now, what happens if LLMs have the same legal standing as humans and are thus able to participate in the economy in the same manner?
- zbentley 6mo ago
  If a linter insists on a weird line of code, I’m probably commenting that line as “recommended by whatever-linter”, yes.
  - martin-t 6mo ago
    I wouldn't but I can see why some people would. I can't point out where I draw the line clearly but here's one different I notice: A recommendation can be both a thing and an action. A piece of text is a recommendation and it does not matter how it was created. Assistance implies some parity in capabilities and cooperative work. Also it can pretty much only be an action, you cannot say "here is some assistance" and point to a thing.

ninjagoo 6mo ago

> Just like stealing fractional amounts of money[3] should not be legal, violating the licenses of the training data by reusing fractional amounts from each should not be legal either. I think you'll find that this is not settled in the courts, depending on how the data was obtained. If the data was obtained legally, say a purchased book, courts have been finding that using it for training is fair use (Bartz v. Anthropic, Kadrey v. Meta). Morally the case gets interesting. Historically, there was no such thing as copyright. The English 1710 Statute of Anne establishing copyright as a public law was titled 'for the Encouragement of Learning' and the US Constitution said 'Congress may secure exclusive rights to promote the progress of science and useful arts'; so essentially public benefits driven by the grant of private benefits. The Moral Bottomline: if you didn't have to eat, would you care about who copies your work as long as you get credited? The more the people that copy your work with attribution, the more famous you'll be. Now that's the currency of the future*. [1] You'll do it for the kudos. [2][3] *Post-Scarcity Future. [1] https://en.wikipedia.org/wiki/Post-scarcity [2] https://en.wikipedia.org/wiki/The_Quiet_War, et. al. [3] https://en.wikipedia.org/wiki/Accelerando

martin-t 6mo ago
> The Moral Bottomline: if you didn't have to eat, would you care about who copies your work as long as you get credited? Yes. I have 2 issues with "post-scarcity": - It often implicitly assumes humanity is one homogeneous group where this state applies to everyone. In reality, if post-scarcity is possible, some people will be lucky enough to have the means to live that lifestyle while others will still by dying of hunger, exposure and preventable diseases. All else being equal, I'd prefer being in the first group and my chance for that is being economically relevant. - It often ignores that some people are OK with having enough while others have a need to have more than others, no matter how much they already have. The second group is the largest cause of exploitation and suffering in the world. And the second group will continue existing in a post-scarcity world and will work hard to make scarcity a real thing again. --- Back to your question: I made the mistake of publishing most of my public code under GPL or AGPL. I regret is because even though my work has brought many people some joy and a bit of my work was perhaps even useful, it has also been used by people who actively enjoy hurting others, who have caused measurable harm and who will continue causing harm as long as they're able to - in a small part enabled by my code. Permissive licenses are socially agnostic - you can use the work and build on top of it no matter who you are and for what purpose. A(GPL) is weakly pro-social - you can use the work no matter what but you can only build on top of it if you give back - this produces some small but non-zero social pressure (enforced by violence through governments) which benefits those who prefer cooperation instead of competition. What I want is a strongly pro-social license - you can use or build on top of my work only if you fulfill criteria I specify such as being a net social good, not having committed any serious offenses, not taking actions to restrict other people's rights without a valid reason, etc. There have been attempts in this direction[0] but not very successful. In a world without LLMs, I'd be writing code using such a license but more clearly specified, even if I had to write my own. Yes, a layer would do a better job, that does not mean anything written by a non-lawyer is completely unenforceable. With LLMs, I have stopped writing public code at all because the way I see it, it just makes people much richer than me even richer at a much faster rate than I can ever achieve myself. Ir just makes inequality worse. And with inequality, exploitation and oppression tends to soon follow. [0]: https://json.org/license.html https://json.org/license.html

deleted 6mo ago

[deleted]

user34283 6mo ago

For [0], it was supposedly shown to do it when specifically prompted to do so. Despite agentic tools being used by millions of developers now, I am not aware of a single real case where accidental reproduction of copyrightable code has been an issue. Further, some model providers offer indemnity clauses. It seems like a non-issue to me, practically.

williamcotton 6mo ago

"Just a few weeks ago a SOTA model was shown to reproduce non-trivial amounts of licensed code[0]." That LLM response is describing a specific project with full attribution.

martin-t 6mo ago
And it proves the code is stored (in a compressed form) in the model.
- williamcotton 6mo ago
  So what's the legal issue here? How does the chardet achieve this? Explain in detail, with shortened code excerpts from the library itself if helpful to the explanation. The prompt is explicitly requesting the source!
  - martin-t 6mo ago
    You're saying the same thing like it's relevant. What matters to me is that the source code is clearly stored encoded/compressed in the model. That to me means the license of the code should apply to the model. Now we just need somebody to ask it to decompress ("reproduce") AGPL code and its owners will have to release it under the AGPL as well. Yes, this is sarcasm but that's how it should work. These models cannot be created without ingesting vast amounts of code created by human labor. The proponents of this kind of stealing argue that the models "generalize" and that somehow makes it right and legal. And here we have counterexample to their generalization argument. Doesn't change the fact labor theft is still labor theft but it might make a difference to the letter of the law, hopefully. I say hopefully because I believe all people who contribute to technical progress should be compensated according to their amount of work and skill level. But of course many "temporarily embarrassed trillionaires" disagree. I don't understand why but I am happy to know that if the tides turn they will be perfectly A-OK with me taking their work without credit or compensation.