porterbaseball

3 ms·

1. 1.▲
   by porterbaseball 5mo ago

   I was curious about the variance of the output and I made some runs w/ deepseek v4 flash and found that it was pretty high? There's also a possibility of strong model memorization on the tasks, I saw a header w/ the authors g
2. 2.▲
   ProgramBench (Meta) Repro: variance across runs and findings(nickcheng0921.github.io)

   2 points by porterbaseball 5mo ago|1 comments
3. 3.▲
   by porterbaseball 5mo ago

   The original ProgramBench paper released by Meta (5/5/26) used single runs for their tasks during benchmarking. I was curious about the variance of the output and I made some runs w/ deepseek v4 flash and found that it was pr