> it's also not very fun Pish-posh! It's a single simple search-and-replace: replace '\n\n\n' with '\n<|endoftext|>\n' or so. For bonus, you can use regexp cap

> it's also not very fun Pish-posh! It's a single simple search-and-replace: replace '\n\n\n' with '\n<|endoftext|>\n' or so. For bonus, you can use regexp capture groups to rewrite the metadata simultaneously - something like '\n\n\n\(.\)\n\(.\)' → '\n<|endoftext|>\n"\1", by \2\n'. > The goal of the post was to generate Beatles lyrics not song lyrics in general. To that end, I'd like to see what you get if you first fine tune on RapGenius to learn general things like song structure, rhyme, etc, then fine tune even further on the Beatles corpus. I suspect you'd get much nicer, less memorized songs. You can do it either way: either train a single model on a multi-artist corpus and then simply prompt it appropriately, or train the single model and then further finetune on just the specific artist. I've tried both in various ways with GPT-2 and StyleGAN, and it's not clear which is best, although I hypothesize that the two-stage pretraining works best with very small corpuses, where in the multi-artist corpus single model, all the other artists might 'squeeze out' the desired artist (a kind of class imbalance), eliminating the transfer benefits. with StyleGAN, a major benefit of the two-stage pretraining approach is that there's no easy way to 'condition' on a specific class or input; so with my anime face generator (https://www.gwern.net/Faces https://www.gwern.net/Faces), when I wanted specific characters, I'd just finetune on that character alone because it's easy to select out just their data and create character-specific corpuses.