3 ms·
There are plenty of people with access to GPT-4 through the API. They can benchmark it however they want.
by sebzim4500 4y ago
There are plenty of people with access to GPT-4 through the API. They can benchmark it however they want.
- mdaniel 4y agoYour suggestion made me wonder: are the OpenAI API responses deterministic? I would expect that if one gets a different reply to the same prompt each time (as I would expect, given the probabilistic nature of LLM), that makes "benchmarking" kind of weird. Or maybe the solution is just to run the prompt 100 times and mean/average across whatever metric is being used