3 ms·

I'm kind of confused - you've claimed that Google's AI was broken, but cite an anecdote over ChatGPT? Regardless, even if these cases did happen often enough,

I'm kind of confused - you've claimed that Google's AI was broken, but cite an anecdote over ChatGPT? Regardless, even if these cases did happen often enough, it's erroneous to assume that this is something universal (i.e. can be applied to all generative AI models) or intentional. Said models are vastly smaller than the sizes of their training datasets, so it's more or less impossible for all the data to be stored verbatim. Some aspects may appear to look like memorization if the piece of data reappears many times in the dataset - in those cases, the algorithm has a really strong incentive to recite that data. This reduces the effectiveness and is considered an artifact that needs to be corrected, not some underlying idea of AI. The way these algorithms work is public knowledge, there's not really any black boxes in the hands of OpenAI that would be relevant here. Considering that, I'm really doubtful that these claims can be easily supported.

jncfhnb 3y ago
Googles research on chat gpt
numpad0 3y ago
There has been successful prompt engineering attempts to force GPT-based models to regurgitate original dataset texts texts texts texts texts texts as for example Wikipedia is a free-content online encyclopedia, written and maintained by a community of volunteers, collectively known as Wikipedians, through open colla--
junofan 3y ago
I think you’re a bit confused—these models are trained by copying works, and the memorization is the incontrovertible proof. Imagine if you download a font and use it to make a logo without paying to license the font.
kmeisthax 3y ago
Google does research on both their own and other people's models all the time. Here's the blog post and paper if you want to know more: https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html https://not-just-memorization.github.io/extracting-training-... You appear to be refuting a slightly different point, though. When OpenAI was making Dall-E 2, they found that duplicates in the training set would incentivize memorization of specific images, as you said. This is more like finding a secret cheat code that would make the model regurgitate everything it had remembered, regardless of how "incentivized" it is to do so or even if it had been aligned to not do that. My personal argument is this: the primary metric that the training process attempts to minimize is perplexity. This is how good the model is at guessing the training set data. Base models are specifically being designed to compress huge amounts of text and we just so happen to accidentally get a decent word calculator out of it. The alignment fine-tuning that happens later adjusts the model to prefer answering questions, but the underlying memorized data is still there. >The way these algorithms work is public knowledge, there's not really any black boxes in the hands of OpenAI that would be relevant here. Nope. Modern GPT is entirely a black box. OpenAI stopped publishing model weights the moment Elon Musk stopped writing the checks. Hell, they don't even publish the model architecture anymore. How GPT-4 works, even on a basic "this is how many transformer layers and attention heads we're using" basis, is a trade secret. Even if you have model weights, the actual meaning of the learned model parameters has never been known; there is active research on figuring them out. One particular problem is polysemanticity. If you look inside a particular hidden layer, you won't see a single "dog" or "cat" neuron in its hidden layers. You'll have a 512-dimension concept bouillabaisse with "dog", "cat", "bird", "guinea pig", "kangaroo", and so on all floating around whatever shape made sense at training time (even if it implies absurdities like "desk is the opposite of loin cloth"). To untangle this, you have to train another AI to pick out monosemantic clusters of neurons that can then be inspected, and that requires extreme amounts of GPU resources.

jncfhnb 3y ago

Googles research on chat gpt

numpad0 3y ago

There has been successful prompt engineering attempts to force GPT-based models to regurgitate original dataset texts texts texts texts texts texts as for example Wikipedia is a free-content online encyclopedia, written and maintained by a community of volunteers, collectively known as Wikipedians, through open colla--

junofan 3y ago

I think you’re a bit confused—these models are trained by copying works, and the memorization is the incontrovertible proof. Imagine if you download a font and use it to make a logo without paying to license the font.

kmeisthax 3y ago

Google does research on both their own and other people's models all the time. Here's the blog post and paper if you want to know more: https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html https://not-just-memorization.github.io/extracting-training-... You appear to be refuting a slightly different point, though. When OpenAI was making Dall-E 2, they found that duplicates in the training set would incentivize memorization of specific images, as you said. This is more like finding a secret cheat code that would make the model regurgitate everything it had remembered, regardless of how "incentivized" it is to do so or even if it had been aligned to not do that. My personal argument is this: the primary metric that the training process attempts to minimize is perplexity. This is how good the model is at guessing the training set data. Base models are specifically being designed to compress huge amounts of text and we just so happen to accidentally get a decent word calculator out of it. The alignment fine-tuning that happens later adjusts the model to prefer answering questions, but the underlying memorized data is still there. >The way these algorithms work is public knowledge, there's not really any black boxes in the hands of OpenAI that would be relevant here. Nope. Modern GPT is entirely a black box. OpenAI stopped publishing model weights the moment Elon Musk stopped writing the checks. Hell, they don't even publish the model architecture anymore. How GPT-4 works, even on a basic "this is how many transformer layers and attention heads we're using" basis, is a trade secret. Even if you have model weights, the actual meaning of the learned model parameters has never been known; there is active research on figuring them out. One particular problem is polysemanticity. If you look inside a particular hidden layer, you won't see a single "dog" or "cat" neuron in its hidden layers. You'll have a 512-dimension concept bouillabaisse with "dog", "cat", "bird", "guinea pig", "kangaroo", and so on all floating around whatever shape made sense at training time (even if it implies absurdities like "desk is the opposite of loin cloth"). To untangle this, you have to train another AI to pick out monosemantic clusters of neurons that can then be inspected, and that requires extreme amounts of GPU resources.