3 ms·
If you are talking about Figure 1 of https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426, you are reading it wrong. Y axis is bits-per-byte and lo
by sanxiyn 4y ago
If you are talking about Figure 1 of https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426, you are reading it wrong. Y axis is bits-per-byte and lower is better.
- visarga 4y agoI saw the graph. Bits-per-byte is not clear for me (for example lower loss does not always mean better accuracy when you train a classifier). What I needed is to know if the generated text has the same "literary" qualities, and also if it can compete in few shot mode on as many tasks as GPT-3 can do. In other words, can we rely on it instead of GPT-3 in realistic scenarios or is it good only for low BpB?
- ShamelessC 4y agoI imagine that GPT-3's ability to complete prompts it hasn't seen will make it quite a bit more versatile than using the Pile. It's the interpolation between the data points that makes deep learning so powerful. This obviously still has plenty of use cases. Could get a batch of similar text to calculate statistics on. Or perhaps augment existing data for training something else. One realistic use case would be to simply present a search engine interface enabling you to find "interesting" text snippets alongside metadata like book author/title matching your description, perhaps for fiction enthusiasts or what have you.