Apple's study suggested a 0.3% drop in benchmark results for GPT-4o from fuzzying the prompts, which to me really means no drop and actually shows the benchmark

Apple's study suggested a 0.3% drop in benchmark results for GPT-4o from fuzzying the prompts, which to me really means no drop and actually shows the benchmark results are more robust than I'd previously expect. It also only looked at GPT-4o, ignoring the allegedly more expensive GPT-4. Finally, I can't help but notice the irony of an article about LLMs that likes to "delve" into "crucial" points "to ensure"... and repeatedly defines the "large language models (LLMs)" acronym from the very first paragraph to the conclusion. It also claims the study was led by Mehrdad Farajtabar when he's listed as the sixth and last author on the actual pre-print: https://arxiv.org/pdf/2410.05229 https://arxiv.org/pdf/2410.05229