3 ms·
Orca: Progressive Learning from Complex Explanation Traces of GPT-4
- ttul 3y agoA decent walkthrough of the paper is here: https://youtu.be/Dt_UNg7Mchg https://youtu.be/Dt_UNg7Mchg I’m surprised this hasn’t received more comments. Orca is an important result. An 11B-parameter model has reached near-parity with a 175B-parameter model on reasoning tasks. You don’t have to squint hard to see where this work is headed.
- ryneandal 3y agoThis is where I heard of Orca last night. Love his breakdowns. The idea (and model) is very exciting. I figured efficiency was going to be the next area of breakthrough, since monolith LLMs seem to be VRAM-limited these days. I hope this is a sign of things to come and ML becomes more democratized!
- parentheses 3y agoThis is fantastic and doing some BOTE math on the cost of training, it's under 10k (160 hours of 20 GPUs at ~$1 per GPU-hour puts us at $3200) so assuming you have a dataset prepared and ready to go, this can be a really huge leap in open source and closed source models!