2 ms·
Your suggestion made me wonder: are the OpenAI API responses deterministic? I would expect that if one gets a different reply to the same prompt each time (as I
by mdaniel 4y ago
Your suggestion made me wonder: are the OpenAI API responses deterministic? I would expect that if one gets a different reply to the same prompt each time (as I would expect, given the probabilistic nature of LLM), that makes "benchmarking" kind of weird. Or maybe the solution is just to run the prompt 100 times and mean/average across whatever metric is being used