4 ms·

So I would want to include a big corpus like GPT-3 or this newfangled "Neo" thing but still have it trained to respond to our own customers based on 200k email

So I would want to include a big corpus like GPT-3 or this newfangled "Neo" thing but still have it trained to respond to our own customers based on 200k email passages. How to create a hybrid?

stellaathena 6y ago
200k emails is not enough to train a model from scratch. If you check out the google colab file in the GPT-Neo repository, it talks about how to fine-tune the model on data which is what you want to do
minimaxir 6y ago
You fine-tune an existing pretrained model on your proprietary dataset.
jalammar 6y ago
I wouldn't trust any model to generate text for customers yet. Not even the largest GPT3. There are no guarantees on what they will output and could be damaging to your business. You're better off either: 1- Defining common "intents" that a lot of customer queries are categorized into, and having a model map the incoming message to the appropriate canned response. Look at Rasa, for an example of this. 2- if you insist on generating the text, have it be a recommendation to a human agent that either chooses to send it or writes their own response.
- flemhans 6y ago
  Thanks for the advice.
kristoo 6y ago
You might get some really promising results with finetuning. If anything, you could build writing assistance that almost automates responses. I've been co-authoring a library that lets you finetune such models in a single line of code. https://github.com/backprop-ai/backprop https://github.com/backprop-ai/backprop In specific the text generation finetuning example should be what you are looking for: https://github.com/backprop-ai/backprop/blob/main/examples/Finetuning_GettingStarted.ipynb https://github.com/backprop-ai/backprop/blob/main/examples/F... Hope this helps, happy to chat more about it. Pretty curious about the results.

4 ms·

So I would want to include a big corpus like GPT-3 or this newfangled "Neo" thing but still have it trained to respond to our own customers based on 200k email

by flemhans 6y ago

So I would want to include a big corpus like GPT-3 or this newfangled "Neo" thing but still have it trained to respond to our own customers based on 200k email passages. How to create a hybrid?

stellaathena 6y ago
200k emails is not enough to train a model from scratch. If you check out the google colab file in the GPT-Neo repository, it talks about how to fine-tune the model on data which is what you want to do
minimaxir 6y ago
You fine-tune an existing pretrained model on your proprietary dataset.
jalammar 6y ago
I wouldn't trust any model to generate text for customers yet. Not even the largest GPT3. There are no guarantees on what they will output and could be damaging to your business. You're better off either: 1- Defining common "intents" that a lot of customer queries are categorized into, and having a model map the incoming message to the appropriate canned response. Look at Rasa, for an example of this. 2- if you insist on generating the text, have it be a recommendation to a human agent that either chooses to send it or writes their own response.
- flemhans 6y ago
  Thanks for the advice.
kristoo 6y ago
You might get some really promising results with finetuning. If anything, you could build writing assistance that almost automates responses. I've been co-authoring a library that lets you finetune such models in a single line of code. https://github.com/backprop-ai/backprop https://github.com/backprop-ai/backprop In specific the text generation finetuning example should be what you are looking for: https://github.com/backprop-ai/backprop/blob/main/examples/Finetuning_GettingStarted.ipynb https://github.com/backprop-ai/backprop/blob/main/examples/F... Hope this helps, happy to chat more about it. Pretty curious about the results.