4 ms·

Yes, it's a very interesting question! This how many language models work. You might be interested in reading about a metric called perplexity. [0] The major c

Yes, it's a very interesting question! This how many language models work. You might be interested in reading about a metric called perplexity. [0] The major caveat is that it's invalid to evaluate on completed, polished work, and then extrapolate to what's practically a napkin note. Another important note about it is that some next token predictions are harder than others. For example, try predicting the next word (the _____) in each of these: > I really love this Dr Seuss Book called "The Cat in the _____ versus > I really love this Dr Seuss book called "The Cat in the Hat." _______ [0] https://en.wikipedia.org/wiki/Perplexity https://en.wikipedia.org/wiki/Perplexity

noduerme 5y ago
You could also make a model which always responds with the word "hat". If the second sentence is the "creative" portion, it would then prove 100% accurate on your in-sample data, and the next sentence could be "Hat, hat hat." And then people could say, see, this brilliant AI speaks like Dr Seuss! You could also just string a bunch of random words together and build a model that learned to predict them with total accuracy. Of course if the words are truly random, it would have basically no chance of predicting the next random word from outside the learning set. The conceit really is that some model got enough important, critical contextual information from the partial symphony to be able to predict with greater than random probability the next portion. On a micro level, one note at a time, maybe. But every note further into the unknown is predicated on the previous ones, so on a macro level it's absurd to think it's "predicting" anything.
diskzero 5y ago
Thanks for sharing that term. A score is so much more complex. I need to dive into the approach Playform took. I notice that team is comprised of experts in a variety of fields, who make a lot of editorial decisions on the output of the AI and then decided the style of the output (scherzo, etc.) and where that particular element might be placed in the score. I wonder what would have happened if they just ignored the AI and completed it using their own compositional knowledge?

noduerme 5y ago

You could also make a model which always responds with the word "hat". If the second sentence is the "creative" portion, it would then prove 100% accurate on your in-sample data, and the next sentence could be "Hat, hat hat." And then people could say, see, this brilliant AI speaks like Dr Seuss! You could also just string a bunch of random words together and build a model that learned to predict them with total accuracy. Of course if the words are truly random, it would have basically no chance of predicting the next random word from outside the learning set. The conceit really is that some model got enough important, critical contextual information from the partial symphony to be able to predict with greater than random probability the next portion. On a micro level, one note at a time, maybe. But every note further into the unknown is predicated on the previous ones, so on a macro level it's absurd to think it's "predicting" anything.

diskzero 5y ago

Thanks for sharing that term. A score is so much more complex. I need to dive into the approach Playform took. I notice that team is comprised of experts in a variety of fields, who make a lot of editorial decisions on the output of the AI and then decided the style of the output (scherzo, etc.) and where that particular element might be placed in the score. I wonder what would have happened if they just ignored the AI and completed it using their own compositional knowledge?