4 ms·
Unfortunately, for transformer-based LLMs the magic starts only when they are trained by more that 10^22 TFlops (preferably 10^24) so smaller models might not c
by treprinum 3y ago
Unfortunately, for transformer-based LLMs the magic starts only when they are trained by more that 10^22 TFlops (preferably 10^24) so smaller models might not cut it even for fine-tuned tasks.
- visarga 3y agoAt medium size (13B) Microsoft Orca demonstrated you can trade off size with larger fine-tuning dataset.
- tangjurine 3y agoAny references on this?
- treprinum 3y agohttps://www.youtube.com/watch?v=tVtOevLrt5U https://www.youtube.com/watch?v=tVtOevLrt5U