3 ms·
They do explain their methodology in some detail in the accompanying GitHub repo: https://github.com/lchen001/LLMDrift https://github.com/lchen001/LLMDrift The
by simonw 3y ago
They do explain their methodology in some detail in the accompanying GitHub repo: https://github.com/lchen001/LLMDrift https://github.com/lchen001/LLMDrift
They seem to have been taking to the API directly and requesting the two different model snapshots.
I'm not convinced by their methodology generally. It looks like everything may have been run with temperature 0.1, which I don't think reflects most real-world usage for example.
- deet 3y agoIn looking at the paper's continuous mention of "ChatGPT" and the repo README's statement that "You don't need API keys to get started" .. are we sure they weren't using type of tools to talk to the ChatGPT API (via a session token, etc) vs the OpenAI API? I do agree they talk about the API in the paper a lot but I don't see an exact methods statement that they directly accessed the non-ChatGPT API anywhere, unless I'm missing it,
- simonw 3y agoI think the lack of API key note is because that notebook is the one that renders the charts for the paper.