2 ms·
It doesn't memorize training data except for very repetitive bits. It needs a fresh copy of the paper in the prompt but only 10% fits.
by visarga 4y ago
It doesn't memorize training data except for very repetitive bits. It needs a fresh copy of the paper in the prompt but only 10% fits.
- sebzim4500 4y agoMaking the context 10x longer would 'only' cost about 100x as much compute. Presumably you could take GPT-3 (trained with the normal context) and then finetune it on a comparitively small amount of data with the new context length, so it shouldn't be enormously expensive to train either. Would be interesting to see how well it could 'understand' a paper, or if more layers etc. would be needed.
- visarga 4y agoWell, it depends. Perceiver can handle big context. https://arxiv.org/abs/2103.03206 https://arxiv.org/abs/2103.03206