4 ms·

> For a while I was frustrated at how slow people have been to realize that GPT-3 sucks but lately I am more amused. Generated text was not good before this er

by rytill 5y ago

> For a while I was frustrated at how slow people have been to realize that GPT-3 sucks but lately I am more amused. Generated text was not good before this era of GPT-X. It’s so much better and more interesting to work with now. It will probably keep getting even better and more controllable.

PaulHoule 5y ago
GPT-X is better than RNNs I grant you but people have built mad-lib and rules-based text generation systems that are absolutely great for specific applications in particular domains. (e.g. GPT-X is still a bridesmaid instead of a bride) I think you could do better with RNNs than most people are doing because of structural problems. Usually when people run RNNs for text generation they start out with the inner state of the system at 0 and then start flipping the coin to choose individual letters so you are starting from a very constrained region of the latent space and not sampling it very well. I read a paper where they through out the idea that you ought to add coefficients for the latent state that you train for at the same time you train the network which means the number of coefficients goes up with the number of text samples but they never actually did it and I never found a paper where somebody tried it. I was working on a project where we were developing models based on abstracts of case studies from pubmed as a stand in for clinical notes (certainly real medical notes are very different but you might say that medical notes should look like the abstract) I had the intuition that, as above, the author (and/or the patient) started out with a latent state (e.g. the patient had a disease before coming in) and that we'd get better results if we did something like the above. It looked like a big and high risk project to develop that kind of model so I proposed something different around supervised training of a "magic magic marker" that could highlight certain areas and unsupervised multi-tasks such as "put the punctuation back in when it is taken out" but the client was hopeful that word2vec would be helpful. I am still hopeful that incremental improvements, attacks on structural weaknesses, and appropriate multi-task training ("did the patient die?") would get a lot more out of RNN and CNN models.

by rytill 5y ago

PaulHoule 5y ago

GPT-X is better than RNNs I grant you but people have built mad-lib and rules-based text generation systems that are absolutely great for specific applications in particular domains. (e.g. GPT-X is still a bridesmaid instead of a bride) I think you could do better with RNNs than most people are doing because of structural problems. Usually when people run RNNs for text generation they start out with the inner state of the system at 0 and then start flipping the coin to choose individual letters so you are starting from a very constrained region of the latent space and not sampling it very well. I read a paper where they through out the idea that you ought to add coefficients for the latent state that you train for at the same time you train the network which means the number of coefficients goes up with the number of text samples but they never actually did it and I never found a paper where somebody tried it. I was working on a project where we were developing models based on abstracts of case studies from pubmed as a stand in for clinical notes (certainly real medical notes are very different but you might say that medical notes should look like the abstract) I had the intuition that, as above, the author (and/or the patient) started out with a latent state (e.g. the patient had a disease before coming in) and that we'd get better results if we did something like the above. It looked like a big and high risk project to develop that kind of model so I proposed something different around supervised training of a "magic magic marker" that could highlight certain areas and unsupervised multi-tasks such as "put the punctuation back in when it is taken out" but the client was hopeful that word2vec would be helpful. I am still hopeful that incremental improvements, attacks on structural weaknesses, and appropriate multi-task training ("did the patient die?") would get a lot more out of RNN and CNN models.