3 ms·
So it's trained on the SWE Bench Pro evalset
by OsrsNeedsf2P 4mo ago
So it's trained on the SWE Bench Pro evalset
- lemonish97 4mo agoWhat is your evidence for this claim?
- fooker 4mo agoThey say hill climbing https://microsoft.ai/news/building-a-hillclimbing-machine-launching-seven-new-mai-models/ https://microsoft.ai/news/building-a-hillclimbing-machine-la... Unless they specifically clarify that the testing and training benchmarks are completely separate, we have to assume they test on the same 'hill' the model climbs.
- ajyoon 4mo ago[flagged]
- jongalloway2 4mo ago[dead]
- artemisart 4mo agoHill climbing doesn't mean much but absolutely doesn't imply they cheat on benchmarks. They have more details here https://microsoft.ai/news/introducing-mai-thinking-1/ https://microsoft.ai/news/introducing-mai-thinking-1/ it seems to be "RL on everything".
- deleted 4mo ago[deleted]
- topsycatt 4mo agoThat's not accurate. Take a look at the paper to see what it is trained on! And specifically decontamination is called out in A.4 https://microsoft.ai/wp-content/uploads/2026/06/main_20260602_2.pdf https://microsoft.ai/wp-content/uploads/2026/06/main_2026060...