3 ms·
nice work! generating good example data is the most important part of finetuning. imo summarization is also a fairly simple task -- I wouldn't be surprised if
by huac 3y ago
nice work! generating good example data is the most important part of finetuning.
imo summarization is also a fairly simple task -- I wouldn't be surprised if a fine-tuned open source model (eg llama 13 / mistral 7) would get to similar performance.
- jxnlco 3y agofor sure! the one thing i was surprised by was how little data gpt3.5 needed, could love for a company to try how the scaling laws work for those smaller models.
- robbomacrae 3y agoI find that bart large 410m [0] parameters does a fine job at summarizing. In Summer AI I alternate between using a copy of that bart large getting hyper-trained on feedback and Chat GPT 3.3 and honestly I don't have a preference between the results. However, thanks to this article I might revisit the summarization techniques used a try a fine tuned 3.5. It would be great to see these techniques compared to Chat GPT 4 Turbo. [0]: https://huggingface.co/facebook/bart-large-cnn https://huggingface.co/facebook/bart-large-cnn
- gwern 3y agoSummarization is a fundamental capability these models develop early on. Remember that one of the things that impressed people about GPT-2 was the discovery that you could prompt them for summaries just by appending "tl;dr" to some text? The issue was always quality, length of summarized text, and control of exactly what kind of summary.