4 ms·
Lessons from the trenches on reproducible evaluation of language models
- soloist11 2y ago[dead]
- jerpint 2y agoOne point they don’t seem to spend much time on is also the difficulty in reproducing outputs in closed-source models. Setting temperature to 0 and setting seeds doesn’t always seem to be enough to get exactly the same results for a given prompt
- 3abiton 2y agoAre there other parameters that affect the output?
- lumost 2y agoLibrary versions with slightly different numeric rounding errors, alternative implementation, runtimes, and hardware variation could all lead to reproduction challenges. There is no obligation for tf’s sigmoid implementation to exactly match PyTorch’s output. The same is true for nvidia vs amd.
- deleted 2y ago[deleted]