5 ms·

This misses that key point that all this prediction can give rise to what looks like astonishing human-level creativity and across many genres. The last decade

This misses that key point that all this prediction can give rise to what looks like astonishing human-level creativity and across many genres. The last decade and half have shown us that with enough data we can pick out patterns well enough to be able to "categorize". But to create, that seemed like a whole another human level outside the realm of mere prediction. Turns out it isn't. What exactly allows LLMs to have this capacity for flexible mimicry? I mean, I can ask it to generate a rap battle between Eminem and Daniel Dennett about consciousness and it can do it very decently. https://twitter.com/SaiDrGaddam/status/1625750884541169666 https://twitter.com/SaiDrGaddam/status/1625750884541169666 That just seems like it shouldn't be so "effortless". What is in here that makes it possible? edit: OK, the article (which is a wonderful introduction, dont want to seem dismissive) does sort of address this but is buried "We can think of the construction of computational language—and semantic grammar—as representing a kind of ultimate compression in representing things. Because it allows us to talk about the essence of what’s possible, without, for example, dealing with all the “turns of phrase” that exist in ordinary human language. And we can view the great strength of ChatGPT as being something a bit similar: because it too has in a sense “drilled through” to the point where it can “put language together in a semantically meaningful way” without concern for different possible turns of phrase." That for me is the real fascinating question. What's allowing LLMs to tap into this semantic grammar and do it so flexibly?

quickthrower2 4y ago
Can it be the next Eminem tho?
- noduerme 4y ago
  what's astonishingly good about this remark is how well it encapsulates the fact that we've been strip-mining and rehashing genuinely new material into sequels for the better part of 50 years
- piersj225 4y ago
  Someone on Reddit has challenged it to a rap battle https://old.reddit.com/r/ChatGPT/comments/10zfvc7/chat_gpt_rap_battled_me https://old.reddit.com/r/ChatGPT/comments/10zfvc7/chat_gpt_r...
- ChaitanyaSai 4y ago
  The real question is how long will it take :) There's something fascinating here about art that is informed and made invaluable by a unique perspective and trajectory or prior experiences. What happens when one beast of a model can have all of those experiences?
corbulo 4y ago
Is creation possible from a 2D semantic abacus? Is that how we want to define creation? If so, creation was already a non-human thing before LLMs by randomizing a dictionary enough times. Or do we tie creation to some kind of value? How should we define it?
taejavu 4y ago
I can’t answer your question but I would push back on the claim that it’s done the requested task very decently. There is nothing uniquely Eminem or Dennett about their respective parts. Eminem has never released a verse with as simplistic rhyme scheme as what’s been produced. Part of the mystique in your question is by assuming that’s it’s done the Eminem/Dennett part of the request justice, when it could really be anyone else’s name attached. I’m impressed that it can create a rap battle about consciousness, but I don’t think it’s done much more than that.
- stevenhuang 4y ago
  You can ask chatgpt for its analysis on what makes it have elements that are uniquely Eminem or dennett. Would be interesting to see what it says.
- mxkopy 4y ago
  Once (if?) the chatGPT servers are up I'm going to ask for a rap battle between 42 Dugg and Daniel Dennett to test exactly this hypothesis. I'm confident that the output will be stylistically different but maybe only superficially. After all some of the first generative NNs that made waves were image style transfer models, and they're fairly small. Who's to say chatGPT can't do a natural language equivalent of the same?
- ChaitanyaSai 4y ago
  Fair point. I'd still argue that we are debating that it can 'merely' do a rap battle about consciousness is amazing. I think back to the time AlexNet came out. Would not have guessed this was possible in ten years. Also, with Eminem, I wonder how much of it having to skirt obscenities etc. As a tune and rhyme deaf person, what would you say is a good example of Eminem's rhyme scheme. Thanks!
  - taejavu 4y ago
    This video is the clearest explanation of advanced rhyme patterns I’ve seen: https://youtu.be/QWveXdj6oZU https://youtu.be/QWveXdj6oZU
somenameforme 4y ago
I'd argue that the reason you're probably impressed with this is because it's outside your domain. You are probably less than impressed that it can generate entire code segments, or fluently answers arbitrary questions because it's easy to see how a training on Stack Exchange and other such sources can easily generate such output. By contrast somebody into rap, but with little knowledge of programming, would probably be floored at the LLM 'being a programmer.' Now go search for 'rap battle example' or other such searches and find the web has even more such content, but you've probably never once in your life seen even a single example of it. And in fact it's also likely been trained on the entire history of every single song, rap, poem, etc. So it's just doing the same thing, but in a field outside your knowledge.
- ChaitanyaSai 4y ago
  That could be. As a programmer I am still very impressed by its ability to create a chrome extension for a very specific use-case that I imagine there is not much data on. This rap battle seem like a very similar cross-genre fusion. It would be interesting to see some study on how well LLMs do transfer learning. Do the meta-patterns learned from code segments (what Wolfram refers to as semantic grammar) be used to pick up meta-patterns from a completely different genre (rap battles, medical literature, legalese etc.) with very few examples. It does seem like it is doing such transfer learning, but yes tough to say if there's say without knowing what data it actually had access to. Seeing an open-source replication that also analyzes how well it does across genres and training data size from that genre would be nice.
joenot443 4y ago
Do you listen to much Eminem? Imagining that in his voice made me laugh, there’s just no way he’d come up with something so bad. It’s similar to the shader code it produces; passable to the naked eye, but upon any level of scrutiny, completely falls apart.

quickthrower2 4y ago

Can it be the next Eminem tho?

noduerme 4y ago
what's astonishingly good about this remark is how well it encapsulates the fact that we've been strip-mining and rehashing genuinely new material into sequels for the better part of 50 years
piersj225 4y ago
Someone on Reddit has challenged it to a rap battle https://old.reddit.com/r/ChatGPT/comments/10zfvc7/chat_gpt_rap_battled_me https://old.reddit.com/r/ChatGPT/comments/10zfvc7/chat_gpt_r...
ChaitanyaSai 4y ago
The real question is how long will it take :) There's something fascinating here about art that is informed and made invaluable by a unique perspective and trajectory or prior experiences. What happens when one beast of a model can have all of those experiences?

corbulo 4y ago

Is creation possible from a 2D semantic abacus? Is that how we want to define creation? If so, creation was already a non-human thing before LLMs by randomizing a dictionary enough times. Or do we tie creation to some kind of value? How should we define it?

taejavu 4y ago

I can’t answer your question but I would push back on the claim that it’s done the requested task very decently. There is nothing uniquely Eminem or Dennett about their respective parts. Eminem has never released a verse with as simplistic rhyme scheme as what’s been produced. Part of the mystique in your question is by assuming that’s it’s done the Eminem/Dennett part of the request justice, when it could really be anyone else’s name attached. I’m impressed that it can create a rap battle about consciousness, but I don’t think it’s done much more than that.

stevenhuang 4y ago
You can ask chatgpt for its analysis on what makes it have elements that are uniquely Eminem or dennett. Would be interesting to see what it says.
mxkopy 4y ago
Once (if?) the chatGPT servers are up I'm going to ask for a rap battle between 42 Dugg and Daniel Dennett to test exactly this hypothesis. I'm confident that the output will be stylistically different but maybe only superficially. After all some of the first generative NNs that made waves were image style transfer models, and they're fairly small. Who's to say chatGPT can't do a natural language equivalent of the same?
ChaitanyaSai 4y ago
Fair point. I'd still argue that we are debating that it can 'merely' do a rap battle about consciousness is amazing. I think back to the time AlexNet came out. Would not have guessed this was possible in ten years. Also, with Eminem, I wonder how much of it having to skirt obscenities etc. As a tune and rhyme deaf person, what would you say is a good example of Eminem's rhyme scheme. Thanks!
- taejavu 4y ago
  This video is the clearest explanation of advanced rhyme patterns I’ve seen: https://youtu.be/QWveXdj6oZU https://youtu.be/QWveXdj6oZU

somenameforme 4y ago

I'd argue that the reason you're probably impressed with this is because it's outside your domain. You are probably less than impressed that it can generate entire code segments, or fluently answers arbitrary questions because it's easy to see how a training on Stack Exchange and other such sources can easily generate such output. By contrast somebody into rap, but with little knowledge of programming, would probably be floored at the LLM 'being a programmer.' Now go search for 'rap battle example' or other such searches and find the web has even more such content, but you've probably never once in your life seen even a single example of it. And in fact it's also likely been trained on the entire history of every single song, rap, poem, etc. So it's just doing the same thing, but in a field outside your knowledge.

ChaitanyaSai 4y ago
That could be. As a programmer I am still very impressed by its ability to create a chrome extension for a very specific use-case that I imagine there is not much data on. This rap battle seem like a very similar cross-genre fusion. It would be interesting to see some study on how well LLMs do transfer learning. Do the meta-patterns learned from code segments (what Wolfram refers to as semantic grammar) be used to pick up meta-patterns from a completely different genre (rap battles, medical literature, legalese etc.) with very few examples. It does seem like it is doing such transfer learning, but yes tough to say if there's say without knowing what data it actually had access to. Seeing an open-source replication that also analyzes how well it does across genres and training data size from that genre would be nice.

joenot443 4y ago

Do you listen to much Eminem? Imagining that in his voice made me laugh, there’s just no way he’d come up with something so bad. It’s similar to the shader code it produces; passable to the naked eye, but upon any level of scrutiny, completely falls apart.