4 ms·

I've been playing with training different sizes[0] of gpt on my own chat data precisely for this reason. Coincidentally, today I was even planning to publish m

by Tenoke 7y ago

I've been playing with training different sizes[0] of gpt on my own chat data precisely for this reason. Coincidentally, today I was even planning to publish my last post and notebook for training gpt2-1.5b and then chatting to oneself with the model. I left it for tomorrow though.. Maybe a mistake. There is quite a lot you can do and talking to my trained model which is responding to me as me can be real weird at times. It's definitely the most engaged Ive been with gpt while talking to myself. Having said that you seem to train here on very little. Still - cool demo. [0] https://svilentodorov.xyz/blog/gpt-345M-finetune/ https://svilentodorov.xyz/blog/gpt-345M-finetune/

MasterScrat 7y ago
I would be very curious to see your notebook - while this simple approach works well with GPT, we are not getting the results we'd want with a more complex question/answer model that uses GPT2. So I'd love to see your implementation details! > Having said that you seem to train here on very little. The datasets provided in the notebook are really meant to be fallbacks for people who are not willing to use their own chat log data. When training on my own data, I have about 500k messages, which starts being enough to get interesting results. edit: wow, I see you're training on "14M facebook messages", that's impressive - do you actually chat that much?!
- Tenoke 7y ago
  I just pushed it, the blog post (which includes the notebook) is here[0]. It's 14mb of data, not sure how many messages it actually comes down to but FB Messenger has been my main platform for talking to friends for the last decade. 0. https://svilentodorov.xyz/blog/gpt-15b-chat-finetune/ https://svilentodorov.xyz/blog/gpt-15b-chat-finetune/
  - MasterScrat 7y ago
    Awesome :D We've published the third notebook: https://colab.research.google.com/drive/1XYNef9zcHhTjt6kM6ydL9oXTshoRknIV https://colab.research.google.com/drive/1XYNef9zcHhTjt6kM6yd... We would gladly have your feedback on our approach
hug 7y ago
I run a discord which is just a collection of people from my local city, with no real fixed subject or agenda. I trained the 345M GPT-2 against the "general" channel, and then set up a discord bot such that every message has a 2% chance that the past 5 messages will be read in as context, and a couple of sentences spat out in response. It's sometimes very lucid, sometimes insane, but most of all it's very entertaining. As a complete aside, I also tried 'transfer learning' against a huge amount of Marxist literature and then a small amount of erotic fiction, but that experiment hasn't quite worked out. Sexy Robot Marx will have to wait.

by Tenoke 7y ago

MasterScrat 7y ago

I would be very curious to see your notebook - while this simple approach works well with GPT, we are not getting the results we'd want with a more complex question/answer model that uses GPT2. So I'd love to see your implementation details! > Having said that you seem to train here on very little. The datasets provided in the notebook are really meant to be fallbacks for people who are not willing to use their own chat log data. When training on my own data, I have about 500k messages, which starts being enough to get interesting results. edit: wow, I see you're training on "14M facebook messages", that's impressive - do you actually chat that much?!

Tenoke 7y ago
I just pushed it, the blog post (which includes the notebook) is here[0]. It's 14mb of data, not sure how many messages it actually comes down to but FB Messenger has been my main platform for talking to friends for the last decade. 0. https://svilentodorov.xyz/blog/gpt-15b-chat-finetune/ https://svilentodorov.xyz/blog/gpt-15b-chat-finetune/
- MasterScrat 7y ago
  Awesome :D We've published the third notebook: https://colab.research.google.com/drive/1XYNef9zcHhTjt6kM6ydL9oXTshoRknIV https://colab.research.google.com/drive/1XYNef9zcHhTjt6kM6yd... We would gladly have your feedback on our approach

hug 7y ago

I run a discord which is just a collection of people from my local city, with no real fixed subject or agenda. I trained the 345M GPT-2 against the "general" channel, and then set up a discord bot such that every message has a 2% chance that the past 5 messages will be read in as context, and a couple of sentences spat out in response. It's sometimes very lucid, sometimes insane, but most of all it's very entertaining. As a complete aside, I also tried 'transfer learning' against a huge amount of Marxist literature and then a small amount of erotic fiction, but that experiment hasn't quite worked out. Sexy Robot Marx will have to wait.