7 ms·

To the uninitiated these may seem like reasonable results because to the uninitiated hip hop is just a bunch of unintelligible words strung together anyway but

To the uninitiated these may seem like reasonable results because to the uninitiated hip hop is just a bunch of unintelligible words strung together anyway but these lyrics don't make much sense. There's no cohesive stream of thought behind the lines. It's a semirandom juxtaposition of phrases and a great example of one of the biggest limitations of deep learning softwares. Even the best are only able to maintain very limited context. This is not just a matter of scaling up to larger networks. Current methods require exponentially larger networks to achieve linear increases in context handling. So then why can our brains do it? The answer is simply that our brains are more than neural networks.

sweezyjeezy 10y ago
Author here. You would be surprised how much of the input lyrics were just semirandom juxtaposition of phrases. The best hip hop is not this, sure, but some of it certainly is.
- croon 10y ago
  Sure, and most human made music is crap. That's not a diss or putting down your work, which is laudable, but the distinction GP presented I believe stands. It's very hard to produce _great_ content this way, simply because there are a lot more variables and dimensions to writing good music and lyrics.
erikschoster 10y ago
Rap music says something. It's hard not to read this project as just a further disenfranchisement its authors.
- sweezyjeezy 10y ago
  Author here. I'm sad that you think that. I love hip hop, and I know how far away this model is from the best MCs. The model has no idea what words actually mean, it's just spitting out a grammar it has learned. We are nowhere near the point where an AI can formulate an original thought and articulate it in rhyme, and I even called my model a 'parlour trick' in the post. It was just supposed to be fun.
wyager 10y ago
> This is not just a matter of scaling up to larger networks. > The answer is simply that our brains are more than neural networks. At the risk of wasting time arguing against mysticism, there is no evidence for either of these statements. (Well, the latter is technically true, but not in the way I think you mean. There's no particular reason an NN couldn't do anything a brain does.) The only thing we can say with confidence is that the OP's model focuses more on rhyme than content, which is true for a lot of popular rappers as well.
- YeGoblynQueenne 10y ago
  >> There's no particular reason an NN couldn't do anything a brain does. Brains (read: humans) can learn from very few examples and in very little time. Despite that, we learn a rich context that is flexible enough to constantly incorporate new knowledge and general enough to transfer learning across diverse domains. Those are all things that ANNs have proven quite incapable of doing, as has any other technology you might want to think of. You don't need to reach for a mystical explanation, either. Our technology is far, far less advanced than the current hype cycle will have you believe. Thinking that we can reproduce the function of the human brain with computers is what is the real mystical belief.
  - baq 10y ago
    our brain is a 20W meat computer. i'd say that believing it can't be reproduced is quite mystical indeed. it's a matter of time; it'll be a 20MW factory-sized supercomputer at first, but it'll be done. not saying it'll happen in this decade, or the next, but this century must be it, assuming humanity makes it to 2100s.
    - deleted 10y ago
      [deleted]
    - YeGoblynQueenne 10y ago
      So, I don't get this. The fact that the brain uses very little power and yet manages to solve really hard problems means that whatever it's doing is very efficient. The fact that ANNs need terrabytes of data and petaflops of processing power and still can only show rudimentary aptitude in mechanical tasks means they're not very efficient at all. Not that anyone ever called ANNs "efficient" (I'm not talking about backprop- but about iterating through covariance matrices). But if they were as efficient as the brain, they'd now be way, way smarter than us. We know from undergraduate Comp Sci that there are problems that can simply not be solved, except with efficient algorithms. The fact that the brain is doing something terribly efficient is a big hint that whatever it's doing requires it to be that (because evolution hand waving hand waving). ANNs are nothing like that - they're practically brute force. So how then can anyone expect that we're going to solve the hard problems the brain can, with ANNs?
gavinpc 10y ago
I made a similar point in defense of Shakespeare[0], and one reply pointed out a kind of category error on my part: > The samples from all of the examples are nonsense. What's interesting is that they, mostly, follow the form of the original. These examples appear much less nonsensical than those, but that is (I'd hypothesize) because rap is so much more grammatically and rhetorically liberal. It does make me wonder how we would distinguish "true" semantics from (trained) formal imitations, when the latter are growing in sophistication. Still, it's a danger in this kind of article that, to quote again from that thread, > it's a usual tendency of NNs to produce output that looks meaningful to non-experts, yet is complete gibberish to experts. [0] https://news.ycombinator.com/item?id=12338081 https://news.ycombinator.com/item?id=12338081
- sweezyjeezy 10y ago
  Nah, they're less non-sensical because of the beam search sampling approach I took. If you don't do this, they're just as bad as that.
Florin_Andrei 10y ago
> So then why can our brains do it? Because current NNs only simulate, like, less than 1 mm^3 of brain matter. Someone writing lyrics for a song has millions of such tiny networks working concurrently in their brain - and then there are higher-level networks supervising and aggregating the smaller nets, and so on. Current AI NN architectures are flat and have no high-level structure. There's no hierarchy. There's no plurisemantic context spanning large time intervals and logic trees. No workable memory organized short-, mid- and long-term. Etc, etc, etc. We're not even scratching the surface yet.
- nickpsecurity 10y ago
  Further, the brain uses subsystems whose architectures are good at the specific problem their solving with smooth integration that somehow often preserves context. The approach of current ANN's seems to be equivalent of taking visual cortext or other single subsystem then trying to apply it to every other subsystem's area of expertise. It will do interesting things but always fall short since it's architecturally not the right tool for the job. It's why my money is on the humans for the upcoming Starcraft challenge. Like poker, bluffing is integral to the process. AI's have been having a really hard time with that in poker except for constrained games. Starcraft provides enough open-ended opportunities that the AI will get smashed by clever human. Hip hop similarly combines tons of references to pop culture, psychology of what goes with what, coordination of presentation, and established musical tricks. At least half of them ANN's suck at by design.
  - Florin_Andrei 10y ago
    There are two ways the AI could win at Starcraft: 1. The game does not really depend on a larger context. What you see is what you get. The "muscle memory" of a relatively simple ANI could therefore be enough. This is partially in contradiction with what you said above about bluffing, but I feel the contradiction is less than 50%. 2. Simple "muscle memory" strategies should not be enough to win the game, but the ANI's lightning speed reactions and its ability to see the whole game at once are enough to outperform more sophisticated-thinking humans who are slower and have tunnel vision w.r.t. the game. Basically the brute-force approach. I'm not placing bets, and I'm as curious as everyone else as to the result of the contest. I'm just saying - if the AI does win, these are the ways it could do that. I'm using the expression "muscle memory", which is inadequate, because I have no better way to express how current NNs operate. They are dumb at the higher semantic levels. They only become powerful through colossal repetition and reinforcement. Watching current NNs being trained never fails to give me flashbacks from my college days when I was practicing karate. We would go to the dojo, pick a technique, and then repeat it an enormous number of times, to let the sequence sink into muscle memory. I'm sure I still have some (natural) NNs in my brain that still have got that thing down pat - I don't have to think about doing the techniques, they just "execute" on their own. But there's no semantic level here, it's just dumb (but blazing fast) automation.
rajacombinator 10y ago
Not sure what you mean, imo the D-Prime lyrics were pretty good, sometimes very clever due to unusual word choice. It sounds a lot like Illmatic era Nas.
- taejavu 10y ago
  > It sounds a lot like Illmatic era Nas. On a technical level that's not true. Illmatic was full of 3 and 4 syllable rhymes, while the D-Prime example rhymes are almost all 1 syllable. Also Nas is very smooth and fluent. The lyrical themes may be closer to Nas, but in terms of flow D-Prime is more like early hip hop - Sugar Hill Gang, Kurtis Blow, Grandmaster Flash etc.
bbctol 10y ago
Our brains may be more than the current iteration of extremely simple, layer-by-layer neural networks, but there's no reason to think they're something fundamentally different than the NN principle of parallel distributed processing at many nodes.

sweezyjeezy 10y ago

Author here. You would be surprised how much of the input lyrics were just semirandom juxtaposition of phrases. The best hip hop is not this, sure, but some of it certainly is.

croon 10y ago
Sure, and most human made music is crap. That's not a diss or putting down your work, which is laudable, but the distinction GP presented I believe stands. It's very hard to produce _great_ content this way, simply because there are a lot more variables and dimensions to writing good music and lyrics.

erikschoster 10y ago

Rap music says something. It's hard not to read this project as just a further disenfranchisement its authors.

sweezyjeezy 10y ago
Author here. I'm sad that you think that. I love hip hop, and I know how far away this model is from the best MCs. The model has no idea what words actually mean, it's just spitting out a grammar it has learned. We are nowhere near the point where an AI can formulate an original thought and articulate it in rhyme, and I even called my model a 'parlour trick' in the post. It was just supposed to be fun.

wyager 10y ago

> This is not just a matter of scaling up to larger networks. > The answer is simply that our brains are more than neural networks. At the risk of wasting time arguing against mysticism, there is no evidence for either of these statements. (Well, the latter is technically true, but not in the way I think you mean. There's no particular reason an NN couldn't do anything a brain does.) The only thing we can say with confidence is that the OP's model focuses more on rhyme than content, which is true for a lot of popular rappers as well.

YeGoblynQueenne 10y ago
>> There's no particular reason an NN couldn't do anything a brain does. Brains (read: humans) can learn from very few examples and in very little time. Despite that, we learn a rich context that is flexible enough to constantly incorporate new knowledge and general enough to transfer learning across diverse domains. Those are all things that ANNs have proven quite incapable of doing, as has any other technology you might want to think of. You don't need to reach for a mystical explanation, either. Our technology is far, far less advanced than the current hype cycle will have you believe. Thinking that we can reproduce the function of the human brain with computers is what is the real mystical belief.
- baq 10y ago
  our brain is a 20W meat computer. i'd say that believing it can't be reproduced is quite mystical indeed. it's a matter of time; it'll be a 20MW factory-sized supercomputer at first, but it'll be done. not saying it'll happen in this decade, or the next, but this century must be it, assuming humanity makes it to 2100s.
  - deleted 10y ago
    [deleted]
  - YeGoblynQueenne 10y ago
    So, I don't get this. The fact that the brain uses very little power and yet manages to solve really hard problems means that whatever it's doing is very efficient. The fact that ANNs need terrabytes of data and petaflops of processing power and still can only show rudimentary aptitude in mechanical tasks means they're not very efficient at all. Not that anyone ever called ANNs "efficient" (I'm not talking about backprop- but about iterating through covariance matrices). But if they were as efficient as the brain, they'd now be way, way smarter than us. We know from undergraduate Comp Sci that there are problems that can simply not be solved, except with efficient algorithms. The fact that the brain is doing something terribly efficient is a big hint that whatever it's doing requires it to be that (because evolution hand waving hand waving). ANNs are nothing like that - they're practically brute force. So how then can anyone expect that we're going to solve the hard problems the brain can, with ANNs?

gavinpc 10y ago

I made a similar point in defense of Shakespeare[0], and one reply pointed out a kind of category error on my part: > The samples from all of the examples are nonsense. What's interesting is that they, mostly, follow the form of the original. These examples appear much less nonsensical than those, but that is (I'd hypothesize) because rap is so much more grammatically and rhetorically liberal. It does make me wonder how we would distinguish "true" semantics from (trained) formal imitations, when the latter are growing in sophistication. Still, it's a danger in this kind of article that, to quote again from that thread, > it's a usual tendency of NNs to produce output that looks meaningful to non-experts, yet is complete gibberish to experts. [0] https://news.ycombinator.com/item?id=12338081 https://news.ycombinator.com/item?id=12338081

sweezyjeezy 10y ago
Nah, they're less non-sensical because of the beam search sampling approach I took. If you don't do this, they're just as bad as that.

Florin_Andrei 10y ago

> So then why can our brains do it? Because current NNs only simulate, like, less than 1 mm^3 of brain matter. Someone writing lyrics for a song has millions of such tiny networks working concurrently in their brain - and then there are higher-level networks supervising and aggregating the smaller nets, and so on. Current AI NN architectures are flat and have no high-level structure. There's no hierarchy. There's no plurisemantic context spanning large time intervals and logic trees. No workable memory organized short-, mid- and long-term. Etc, etc, etc. We're not even scratching the surface yet.

nickpsecurity 10y ago
Further, the brain uses subsystems whose architectures are good at the specific problem their solving with smooth integration that somehow often preserves context. The approach of current ANN's seems to be equivalent of taking visual cortext or other single subsystem then trying to apply it to every other subsystem's area of expertise. It will do interesting things but always fall short since it's architecturally not the right tool for the job. It's why my money is on the humans for the upcoming Starcraft challenge. Like poker, bluffing is integral to the process. AI's have been having a really hard time with that in poker except for constrained games. Starcraft provides enough open-ended opportunities that the AI will get smashed by clever human. Hip hop similarly combines tons of references to pop culture, psychology of what goes with what, coordination of presentation, and established musical tricks. At least half of them ANN's suck at by design.
- Florin_Andrei 10y ago
  There are two ways the AI could win at Starcraft: 1. The game does not really depend on a larger context. What you see is what you get. The "muscle memory" of a relatively simple ANI could therefore be enough. This is partially in contradiction with what you said above about bluffing, but I feel the contradiction is less than 50%. 2. Simple "muscle memory" strategies should not be enough to win the game, but the ANI's lightning speed reactions and its ability to see the whole game at once are enough to outperform more sophisticated-thinking humans who are slower and have tunnel vision w.r.t. the game. Basically the brute-force approach. I'm not placing bets, and I'm as curious as everyone else as to the result of the contest. I'm just saying - if the AI does win, these are the ways it could do that. I'm using the expression "muscle memory", which is inadequate, because I have no better way to express how current NNs operate. They are dumb at the higher semantic levels. They only become powerful through colossal repetition and reinforcement. Watching current NNs being trained never fails to give me flashbacks from my college days when I was practicing karate. We would go to the dojo, pick a technique, and then repeat it an enormous number of times, to let the sequence sink into muscle memory. I'm sure I still have some (natural) NNs in my brain that still have got that thing down pat - I don't have to think about doing the techniques, they just "execute" on their own. But there's no semantic level here, it's just dumb (but blazing fast) automation.

rajacombinator 10y ago

Not sure what you mean, imo the D-Prime lyrics were pretty good, sometimes very clever due to unusual word choice. It sounds a lot like Illmatic era Nas.

taejavu 10y ago
> It sounds a lot like Illmatic era Nas. On a technical level that's not true. Illmatic was full of 3 and 4 syllable rhymes, while the D-Prime example rhymes are almost all 1 syllable. Also Nas is very smooth and fluent. The lyrical themes may be closer to Nas, but in terms of flow D-Prime is more like early hip hop - Sugar Hill Gang, Kurtis Blow, Grandmaster Flash etc.

bbctol 10y ago

Our brains may be more than the current iteration of extremely simple, layer-by-layer neural networks, but there's no reason to think they're something fundamentally different than the NN principle of parallel distributed processing at many nodes.