4 ms·
The problem with that approach is that is very hard to verify that the output is consistent. If you get pure json, you can check it against a schema, and regula
by BiteCode_dev 4y ago
The problem with that approach is that is very hard to verify that the output is consistent. If you get pure json, you can check it against a schema, and regularly check that chatgpt is not outputting BS (which is does easily). With turing complete code, it's way harder.
- williamcotton 4y agoThe approach I've been using is that I test against a corpus of questions with known answers. Using sample-and-vote, that is asking for some number of variable completions with a non-zero temperature and treating the explicitly computed solutions as votes, has smoothed over issues with basic syntax and logic errors. This is pretty similar to the approach used in the Toolformer paper, other than sample-and-vote, which I believe is novel.
- triyambakam 4y agoIn this context what does temperature mean?
- williamcotton 4y agoTemperature is an option passed into the LLM. A temperature of 0 means the response is deterministic, that is that it always returns the same completion. A temperature of greater than zero results in increasing variation in the completion.