> The model was sparse fine-tuned on GSM8K, a dataset with high-quality and diverse grade school math problems. For this task zero-shot evaluation achieved 0% w

> The model was sparse fine-tuned on GSM8K, a dataset with high-quality and diverse grade school math problems. For this task zero-shot evaluation achieved 0% while 8-shot evaluation achieved 6.8%. The model was therefore fine-tuned via supervised fine-tuning (SFT). That's a heck of a score to gloss over. For reference, the GSM8K dataset has questions like this: > Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? A LLM that can't get above 10% performance 8-shot on that is pretty much worthless by modern standards. (Or at least worthless for math problems, but I haven't been very impressed by MPT in other areas either.) So what the article tells us is that they managed to get garbage much faster than originally. Which .. yeah, that's not great.