3 ms·

Why wouldn’t it be feasible? (Maybe this depends on what you mean by ‘feasible’.) There’s no technical reason you can’t back-track the weights and make a list o

by dahart 4y ago

Why wouldn’t it be feasible? (Maybe this depends on what you mean by ‘feasible’.) There’s no technical reason you can’t back-track the weights and make a list of which tokens from which training data were sampled. The list might be long, it could be impractical, but that has little bearing on whether it’s technically possible, right? The problem here happens when the same source is sampled for many tokens in a row because it’s the only match for the context. It could also happen that many tokens in a row each have a long list of sources, but when put together have a subset of sources that appear in every token’s list. That means that someone’s input is being repeated verbatim, even if the network wasn’t trying to reproduce a single source. We could prune the list of attribution sources at the expense of compute by running largest common subset algorithms, which might be sufficient for attribution tracing? It feels like this whole question might hinge on Fair Use. The network is copying other people’s code one token at a time. We (society & copyright law) all tend to agree that’s fine when it’s a single token out of context, and we all tend to agree it’s not fine when the whole output program matches any single input source. The question naturally becomes, “where’s the line, how many tokens in a row from a single source should be allowed?”

wrs 4y ago
The token “if” appears in a heck of a lot of inputs, and a heck of a lot of outputs. What you’re describing seems like basically building an inverted index of the input corpus and then doing a search for the output. If the answer comes back with a high relevance then you consider it “traced”. I wonder what the size of that inverted index would be compared to the size of the generative model.
- dahart 4y ago
  That sounds exactly right to me, or at least this is one specific way to implement a solution to the problem. Good question on size. Speculating… I would guess the size of the index relates not to the size of the model, but to the size of the training data. If you built an extremely naive and straightforward uncompressed index like this, you could imagine building the complete list of known tokens, and for each one the complete list of pairs of source index and character offset into each source. If a pair of 64 bit ints was sufficient for this, you’d have 128 bits per token for the index, compared to whatever the average size token is (which I’d guess is something like 2-4 bytes uncompressed). If that napkin math is anything close to reality, then I might expect the index to be ~8x the size of the training data (with error bars of like 2x-4x on both sides). Maybe we just call it an order of magnitude?
  - leereeves 4y ago
    I think you'd find that most code matches (some) other existing code. Some code, like the famous fast inverse square root, is widely shared. Even trivial code is often just copy-pasted from a popular SO answer. Other code is driven by something like convergent evolution. It ends up similar to other code because of the limits of well known algorithms, language syntax, APIs, boilerplate, common coding styles, etc. In other words, I doubt most human programmers would pass a "copyright check" against all existing open source code.
    - dahart 4y ago
      If true, how do you think these things should reflect on copyright or AI policies? Does it matter whether “most” human written code meets the same standard that we have for AI, and if so, why? It’s hard to discuss unverifiable claims of “most” code. A lot of code, maybe most, or maybe not, is proprietary and kept within corporate walls, so we have no idea how much is or is not copied. SO answers are allowed to be copied, by definition. (And at least with SO answers, the code is publicly accessible, and attribution tracking after the fact is closer to possible, right?) You do have a good point that attribution for some code could be practically impossible to track. This might get a lot worse if we allow AI to remix and republish it, that could even cause feedback loops if we’re not more careful about tracking attributions.
      - leereeves 4y ago
        Copyright requires an element of creativity and originality. The easiest code snippets for AI to reproduce are things like language syntax (loops, definitions, etc), implementations of common algorithms, and boilerplate, and an index would probably find a lot of matches for those, but are they original or creative? So the challenge for an index would be finding a rare case of creative, important code in a sea of trivial matches. Incidentally, answers on SO are licensed cc-by-sa, so it's easy to violate copyright when copying them; but no one seems to care, suggesting that small code snippets are indeed "trivial".
belorn 4y ago
Fair use is a bit more context dependent, especially when applied on other form of media. Unstable diffusion is a big example where many seems to feel that no fair use should be allowed regardless of how small of a token is taken by the AI. A small amount also depend on the original work and how it fit into society at large. Having fair use to be the pillar that all AI training stand on is going to take a while.
- dahart 4y ago
  Yeah good point, TBH I feel like the most practical way forward right now isn’t to wait for Fair Use to clarify AI, it’s probably going to take training data that’s better curated and partitioned into groups that explicitly allow certain kinds of ML use, and models that aren’t leaky across partitions (which sounds like what the problem is with Copilot in this case). This might mean new kinds of code licenses that specify whether it can be used for ML training or not, and companies are going to need to start respecting those licenses rather than assuming they can train on everything reachable from the internet. Otherwise, if we are going to let companies ignore copyright and licenses because of their pinky promise that their models won’t be recognizable copies of any single input, then we really do need a legal criteria for how much of a single input is fair game, right?
johnfn 4y ago
The failure mode I see here, which seems quite likely, is that attribution would almost always list tens of thousands of source files, or more. They doesn’t seem particularly useful or meaningful.
- dahart 4y ago
  Yeah, correct, I would expect the same thing to happen if you list all attributions. That’s why I mentioned a criteria for filtering by number of tokens in a row for a given attribution, and one possible method for doing the filtering via largest common subsets. While the very long list would be impractical (but not infeasible), sorting and filtering the list by how much each source contributes would make it easier to manage. Pruning all the entries that are below a contribution threshold would almost certainly shrink it to a very small fraction of the total number of attributions. We don’t need to list all the attributions that contribute to only one token, all we need to know is which attributions ended up with 5 or 50 or 500 tokens in a row, right? Likely not many.
nonrandomstring 4y ago
The judge/law will say it comes down to intent. Here's my take from an audio DSP viewpoint. A composer listens to 10,000 hours of music. One day she writes an original piece based on the annealed parameters (temporal, spectral, intensity, pitch, sequence...) of a million artists. However it sounds like another specific artist who sues. It is not a cover, a remix, homage, or even forged in the genre... it's just accidentally a bit too like it. (compare: Banana Splits vs. Bob Marley - and - Huey Lewis vs. Ray Parker Jr. - which completely misses the real impact of "Pop Muzik" by M) The question is, was she exposed to influencing materials incorporated without intent or was it plagiarism (intent to reproduce a derivative etc) ? By contrast I can take a piece of music, break it down by analysis into melodies, chords, timings, and use FFT to extract the precise spectrums of instruments, feed those to a resynthesis engine that finds new synthesiser parameters to create an exact sound-alike and then deliberately recreate a piece "In the near style of artist X". With a little musical processing I can change the key, inversions, re-template the rhythm to a new swing... always pushing the derived piece into new territory until eventually it's barely recognisable. Nonetheless, in the second case I have clearly intended to steal someone's idea and "make it my own" by automatable transformation. In the former case it seems to be a "genuine labour". (whatever that means in 2023) The genuine artist intends to make something through intellectual labour. Maybe that's the real question. I mean, about "labour". If the cost of the labour tends to zero, does it really matter?
c7b 4y ago
> There’s no technical reason you can’t back-track the weights and make a list of which tokens from which training data were sampled. The weights map inputs to outputs, not training samples to outputs. They are adjusted during the training phase and then kept fixed (until there's a new iteration of the model; and let's ignore online learning for simplicity's sake now, the major models we're talking about here don't use it afaik). From the perspective of a user who inputs prompts and receives answers, the weights are constant. We could create some score indicating how much each training sample contributed to the weights overall, but then that score would be the same for all prompts. It's really not clear to me what you mean by 'sampling from training data' for a specific prompt. The best I could imagine would be creating another metric measuring the distance between the prompt and all training samples, and then somehow combining that with the first score to get an overall contribution metric for this prompt. But that would be a) quite a bit of guesswork with a lot of modelling freedom and b) quite different from what you described.
janalsncm 4y ago
The model isn’t sampling from training data during inference. It’s sampling from a latent space created during training, which used all of the training data to create it. Fundamentally, the presence of a next token in some of the samples in the corpus is just as informative as the absence of that token from other samples. You can’t cite one without the other. If you wanted to give attribution you’d need to list the entire training corpus.

by dahart 4y ago

wrs 4y ago

The token “if” appears in a heck of a lot of inputs, and a heck of a lot of outputs. What you’re describing seems like basically building an inverted index of the input corpus and then doing a search for the output. If the answer comes back with a high relevance then you consider it “traced”. I wonder what the size of that inverted index would be compared to the size of the generative model.

dahart 4y ago
That sounds exactly right to me, or at least this is one specific way to implement a solution to the problem. Good question on size. Speculating… I would guess the size of the index relates not to the size of the model, but to the size of the training data. If you built an extremely naive and straightforward uncompressed index like this, you could imagine building the complete list of known tokens, and for each one the complete list of pairs of source index and character offset into each source. If a pair of 64 bit ints was sufficient for this, you’d have 128 bits per token for the index, compared to whatever the average size token is (which I’d guess is something like 2-4 bytes uncompressed). If that napkin math is anything close to reality, then I might expect the index to be ~8x the size of the training data (with error bars of like 2x-4x on both sides). Maybe we just call it an order of magnitude?
- leereeves 4y ago
  I think you'd find that most code matches (some) other existing code. Some code, like the famous fast inverse square root, is widely shared. Even trivial code is often just copy-pasted from a popular SO answer. Other code is driven by something like convergent evolution. It ends up similar to other code because of the limits of well known algorithms, language syntax, APIs, boilerplate, common coding styles, etc. In other words, I doubt most human programmers would pass a "copyright check" against all existing open source code.
  - dahart 4y ago
    If true, how do you think these things should reflect on copyright or AI policies? Does it matter whether “most” human written code meets the same standard that we have for AI, and if so, why? It’s hard to discuss unverifiable claims of “most” code. A lot of code, maybe most, or maybe not, is proprietary and kept within corporate walls, so we have no idea how much is or is not copied. SO answers are allowed to be copied, by definition. (And at least with SO answers, the code is publicly accessible, and attribution tracking after the fact is closer to possible, right?) You do have a good point that attribution for some code could be practically impossible to track. This might get a lot worse if we allow AI to remix and republish it, that could even cause feedback loops if we’re not more careful about tracking attributions.
    - leereeves 4y ago
      Copyright requires an element of creativity and originality. The easiest code snippets for AI to reproduce are things like language syntax (loops, definitions, etc), implementations of common algorithms, and boilerplate, and an index would probably find a lot of matches for those, but are they original or creative? So the challenge for an index would be finding a rare case of creative, important code in a sea of trivial matches. Incidentally, answers on SO are licensed cc-by-sa, so it's easy to violate copyright when copying them; but no one seems to care, suggesting that small code snippets are indeed "trivial".

belorn 4y ago

Fair use is a bit more context dependent, especially when applied on other form of media. Unstable diffusion is a big example where many seems to feel that no fair use should be allowed regardless of how small of a token is taken by the AI. A small amount also depend on the original work and how it fit into society at large. Having fair use to be the pillar that all AI training stand on is going to take a while.

dahart 4y ago
Yeah good point, TBH I feel like the most practical way forward right now isn’t to wait for Fair Use to clarify AI, it’s probably going to take training data that’s better curated and partitioned into groups that explicitly allow certain kinds of ML use, and models that aren’t leaky across partitions (which sounds like what the problem is with Copilot in this case). This might mean new kinds of code licenses that specify whether it can be used for ML training or not, and companies are going to need to start respecting those licenses rather than assuming they can train on everything reachable from the internet. Otherwise, if we are going to let companies ignore copyright and licenses because of their pinky promise that their models won’t be recognizable copies of any single input, then we really do need a legal criteria for how much of a single input is fair game, right?

johnfn 4y ago

The failure mode I see here, which seems quite likely, is that attribution would almost always list tens of thousands of source files, or more. They doesn’t seem particularly useful or meaningful.

dahart 4y ago
Yeah, correct, I would expect the same thing to happen if you list all attributions. That’s why I mentioned a criteria for filtering by number of tokens in a row for a given attribution, and one possible method for doing the filtering via largest common subsets. While the very long list would be impractical (but not infeasible), sorting and filtering the list by how much each source contributes would make it easier to manage. Pruning all the entries that are below a contribution threshold would almost certainly shrink it to a very small fraction of the total number of attributions. We don’t need to list all the attributions that contribute to only one token, all we need to know is which attributions ended up with 5 or 50 or 500 tokens in a row, right? Likely not many.

nonrandomstring 4y ago

The judge/law will say it comes down to intent. Here's my take from an audio DSP viewpoint. A composer listens to 10,000 hours of music. One day she writes an original piece based on the annealed parameters (temporal, spectral, intensity, pitch, sequence...) of a million artists. However it sounds like another specific artist who sues. It is not a cover, a remix, homage, or even forged in the genre... it's just accidentally a bit too like it. (compare: Banana Splits vs. Bob Marley - and - Huey Lewis vs. Ray Parker Jr. - which completely misses the real impact of "Pop Muzik" by M) The question is, was she exposed to influencing materials incorporated without intent or was it plagiarism (intent to reproduce a derivative etc) ? By contrast I can take a piece of music, break it down by analysis into melodies, chords, timings, and use FFT to extract the precise spectrums of instruments, feed those to a resynthesis engine that finds new synthesiser parameters to create an exact sound-alike and then deliberately recreate a piece "In the near style of artist X". With a little musical processing I can change the key, inversions, re-template the rhythm to a new swing... always pushing the derived piece into new territory until eventually it's barely recognisable. Nonetheless, in the second case I have clearly intended to steal someone's idea and "make it my own" by automatable transformation. In the former case it seems to be a "genuine labour". (whatever that means in 2023) The genuine artist intends to make something through intellectual labour. Maybe that's the real question. I mean, about "labour". If the cost of the labour tends to zero, does it really matter?

c7b 4y ago

> There’s no technical reason you can’t back-track the weights and make a list of which tokens from which training data were sampled. The weights map inputs to outputs, not training samples to outputs. They are adjusted during the training phase and then kept fixed (until there's a new iteration of the model; and let's ignore online learning for simplicity's sake now, the major models we're talking about here don't use it afaik). From the perspective of a user who inputs prompts and receives answers, the weights are constant. We could create some score indicating how much each training sample contributed to the weights overall, but then that score would be the same for all prompts. It's really not clear to me what you mean by 'sampling from training data' for a specific prompt. The best I could imagine would be creating another metric measuring the distance between the prompt and all training samples, and then somehow combining that with the first score to get an overall contribution metric for this prompt. But that would be a) quite a bit of guesswork with a lot of modelling freedom and b) quite different from what you described.

janalsncm 4y ago

The model isn’t sampling from training data during inference. It’s sampling from a latent space created during training, which used all of the training data to create it. Fundamentally, the presence of a next token in some of the samples in the corpus is just as informative as the absence of that token from other samples. You can’t cite one without the other. If you wanted to give attribution you’d need to list the entire training corpus.