2 ms·
phi3 mini is surprisingly capable given its size. You can teach small transformers to do stuff well, you just can't have good general purpose small models.
by CuriouslyC 3y ago
phi3 mini is surprisingly capable given its size. You can teach small transformers to do stuff well, you just can't have good general purpose small models.
- jasonjmcghee 3y agoTotally. But they aren't fine tuning these afaict- but comparing general purpose capabilities.
- Der_Einzige 3y agoThe point still stands, Phi3 is an excellent model and shows that good models don’t need that many parameters You should see the work on ReFT coming from mannings group showing that you can instruction fine tune models by modifying like, 0.00001% of the parameters. By doing it this way, you significantly mitigate the risk of catastrophic forgetting.