4 ms·
For prompt adherence it still fails on tasks that Gemma2 27b nails every time. I haven't been impressed with any of the Phi family of models. The large context
by thot_experiment 2y ago
For prompt adherence it still fails on tasks that Gemma2 27b nails every time. I haven't been impressed with any of the Phi family of models. The large context is very nice, though Gemma2 plays very well with self-extend.
- CuriousCosmic 2y agoYeah they mention this in the weaknesses section. > While phi-4 demonstrates relatively strong performance in answering questions and performing reasoning tasks, it is less proficient at rigorously following detailed instructions, particularly those involving specific formatting requirements.
- thot_experiment 2y agoAh good catch, I am forever cursed in my preference for snake over camel.
- impossiblefork 2y agoIt's a much smaller model though. I think the point is more the demonstration that such a small model can have such good performance than any actual usefulness.
- magicalhippo 2y agoGemma2 9B has significantly better prompt adherence than Llama 3.1 8B in my experience. I've just assumed it's down to how it was trained, but no expert.