3 ms·

I spent a week trying to do this by hunting down disparate blog posts that all used a different ML framework or GPT2 finetuning tool. I didn't understand what y

I spent a week trying to do this by hunting down disparate blog posts that all used a different ML framework or GPT2 finetuning tool. I didn't understand what you were supposed to do and ended up with gibberish output instead. For example, isn't the dataset supposed to be specifically formatted to separate significant blocks of text, with something like <!endoftext>? The OP doesn't mention anything about this. It's these kinds of things that make me doubt myself at every turn when doing this "finetuning" thing. I wish there was a clearer tutorial for people only interested in retraining GPT2 to generate different kinds of text, alongside a description of how each step works at an intuitive level. A lot of times the author starts right off with "okay let's start finetuning" without really defining what "finetuning" is or does. I just want to understand the scope of knowledge needed to know how to use my own text corpus as the input source without needing to become an expert in ML. (Maybe this activity isn't for me if that's how I'm going to think about it?) If anything, I wish Talk to Transformer still existed as a downloadable model. I have yet to find another model that generates text like it did. At the end of the day I just sigh and look at all the random article and comment and adventure generators other people have succeeded in making without "getting" it.

serendipityrecs 6y ago
Hi, I don't have a comprehensive guide or anything like that, but I can quickly answer your questions: 1. Delimiting your text with <|endoftext|> lets the code know where the boundaries of your text chunks are. It makes sure that during training you're not inadvertently continuing past the end of a chunk of text into a totally unrelated one. However, it seems like there is a bug with this re how the nsheppard fork works: https://github.com/openai/gpt-2/issues/222 https://github.com/openai/gpt-2/issues/222 Even if you're not doing this, you shouldn't be getting gibberish, so you likely have another issue in your set up. 2. Finetuning just means training on top of a somewhat already trained model. GPT-2 is a language model, which means it's trying to learn (and generates from) a probability distribution over words (actually tokens). That probability distribution is going to be different depending on what corpus you're working with (wikipidia articles vs news vs reddit). GPT-2 (the models you download) is trained on a particular corpus but you as you fine-tune on your own corpus it's going to start pulling the learned probability distribution towards that of your dataset's. I think the best way to play with this stuff is to run the code, like you're doing. When you get something that doesn't make sense, dig into the code/paper. There's also a lot of information in the readmes/issues for these repos.
- nonbirithm 6y ago
  Thank you for the pointers, I'll rethink my strategy.
ogou 6y ago
I spent a month trying to get an NVIDIA Jetson Nano to be a text/image generator. Got lost in dependency hell and just ran out of memory. I found Max Woolf's Google Colab notebook and had something working in a day. Ended up creating this, which generates text without a GPU, although it was trained with one in Colab. https://artreviewgenerator.com/ https://artreviewgenerator.com/ Currently writing a blog post about the process, but it's pretty long.

serendipityrecs 6y ago

Hi, I don't have a comprehensive guide or anything like that, but I can quickly answer your questions: 1. Delimiting your text with <|endoftext|> lets the code know where the boundaries of your text chunks are. It makes sure that during training you're not inadvertently continuing past the end of a chunk of text into a totally unrelated one. However, it seems like there is a bug with this re how the nsheppard fork works: https://github.com/openai/gpt-2/issues/222 https://github.com/openai/gpt-2/issues/222 Even if you're not doing this, you shouldn't be getting gibberish, so you likely have another issue in your set up. 2. Finetuning just means training on top of a somewhat already trained model. GPT-2 is a language model, which means it's trying to learn (and generates from) a probability distribution over words (actually tokens). That probability distribution is going to be different depending on what corpus you're working with (wikipidia articles vs news vs reddit). GPT-2 (the models you download) is trained on a particular corpus but you as you fine-tune on your own corpus it's going to start pulling the learned probability distribution towards that of your dataset's. I think the best way to play with this stuff is to run the code, like you're doing. When you get something that doesn't make sense, dig into the code/paper. There's also a lot of information in the readmes/issues for these repos.

nonbirithm 6y ago
Thank you for the pointers, I'll rethink my strategy.

ogou 6y ago

I spent a month trying to get an NVIDIA Jetson Nano to be a text/image generator. Got lost in dependency hell and just ran out of memory. I found Max Woolf's Google Colab notebook and had something working in a day. Ended up creating this, which generates text without a GPU, although it was trained with one in Colab. https://artreviewgenerator.com/ https://artreviewgenerator.com/ Currently writing a blog post about the process, but it's pretty long.