Yep that is something I've been dumbfounded by as well - the official Microsoft phi4 upload also suffers on MATH so at least we can rule out it's because I did

Yep that is something I've been dumbfounded by as well - the official Microsoft phi4 upload also suffers on MATH so at least we can rule out it's because I did something wrong. I thought of two possibilities: 1. 509 does better on MATH but absolutely terribly on IFEVAL because it does not use a chat template - whilst others so use the chat template. 2. I think HF uses exact matching I think so maybe that's the culprit. I can test 1. by resubmitting without using the chat template!