4 ms·

Right now chatgpt employs humans to correct itself by tagging incorrect responses. Chatgpt has ingested paper or book and git repos.

• visarga 4y ago
  It doesn't memorize training data except for very repetitive bits. It needs a fresh copy of the paper in the prompt but only 10% fits.
  • sebzim4500 4y ago
    Making the context 10x longer would 'only' cost about 100x as much compute. Presumably you could take GPT-3 (trained with the normal context) and then finetune it on a comparitively small amount of data with the new context length, so it shouldn't be enormously expensive to train either. Would be interesting to see how well it could 'understand' a paper, or if more layers etc. would be needed.
    • visarga 4y ago
      Well, it depends. Perceiver can handle big context. https://arxiv.org/abs/2103.03206 https://arxiv.org/abs/2103.03206