4 ms·
50% on HumanEval with just 1.3B model
- asicsp 3y agoDupe: "Textbooks are all you need" https://news.ycombinator.com/item?id=36413768 https://news.ycombinator.com/item?id=36413768
- deleted 3y ago[deleted]
- daniel-cussen 3y ago[dead]
- p0w3n3d 3y agoWhat is B in this context? Billion?
- seanthemon 3y agoYes, nB is n-billion number of parameters
- bmacho 3y agoYes, 10^9.
- bigyikes 3y agoFor reference, this beat GPT-3.5 which scores 47%, but not GPT-4 which scored a massive 67%. Beating out GPT-3.5 at any task with such a small model is very cool to me. How much longer until these dumb virtual assistants (Siri, Google, Alexa) get replaced with on-device LLMs? We’ve gotta be getting close. These small, optimized models are catching up quickly in so many domains.
- RecycledEle 3y agoI wonder if learning to train Generative AIs will teach us anything about teaching humans? I mean other than the use of AI tutors. Can we determine the usefulness of text by how well it trains AI?
- throwaway4good 3y agoWhat is HumanEval?