3 ms·
This is great for speed, maybe we can also increase the window size if the model is so small, but how about the quality of the generated text? With a 20x smalle
by visarga 4y ago
This is great for speed, maybe we can also increase the window size if the model is so small, but how about the quality of the generated text? With a 20x smaller model does quality drop?
How many chunks do you retrieve? The paper shows best results at k=1 and then at k>50.
- sanxiyn 4y agoIf you are talking about Figure 1 of https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426, you are reading it wrong. Y axis is bits-per-byte and lower is better.
- visarga 4y agoI saw the graph. Bits-per-byte is not clear for me (for example lower loss does not always mean better accuracy when you train a classifier). What I needed is to know if the generated text has the same "literary" qualities, and also if it can compete in few shot mode on as many tasks as GPT-3 can do. In other words, can we rely on it instead of GPT-3 in realistic scenarios or is it good only for low BpB?
- ShamelessC 4y agoI imagine that GPT-3's ability to complete prompts it hasn't seen will make it quite a bit more versatile than using the Pile. It's the interpolation between the data points that makes deep learning so powerful. This obviously still has plenty of use cases. Could get a batch of similar text to calculate statistics on. Or perhaps augment existing data for training something else. One realistic use case would be to simply present a search engine interface enabling you to find "interesting" text snippets alongside metadata like book author/title matching your description, perhaps for fiction enthusiasts or what have you.