4 ms·

We found that having pre-determined sections are indeed very useful. One for context, one for rules, one for output format, one for input data, etc. Also, you

We found that having pre-determined sections are indeed very useful. One for context, one for rules, one for output format, one for input data, etc. Also, you can as for a structured output (gpt spits out JSON fine) but you need to be super explicit about each field possible values and the conditions they appear in. Templating prompts with jinja goes a long way for testing. And you will need a lot of testing, if for nothing else than remove variability in answers. It's funny because people keep saying gpt will removing all values from having writer skills, but being a good writer helps a lot with crafting good prompts.

williamcotton 4y ago
I've been writing prompts with the intent of producing JSON that contains JS functions to evaluate. Since the response needs to be parsed and evaluated in a specific structure it means that changes need to be made on the evaluation side as well as the prompt template side. So I've been writing the translation examples (few-shot examples [not my favorite term]) as TypeScript, transpiling to JS and compiling with other examples and a prelude, and using the same method to build the translation examples as to build future prompts. It saves a lot of silly mistakes! JavaScript has less tokens than TypeScript, so it seems more economical to convert to JS beforehand instead of passing TS to the LLM! I wouldn't be surprised if TS resulted in better solutions, though... add it to the endless list of things to test out...
- BiteCode_dev 4y ago
  The problem with that approach is that is very hard to verify that the output is consistent. If you get pure json, you can check it against a schema, and regularly check that chatgpt is not outputting BS (which is does easily). With turing complete code, it's way harder.
  - williamcotton 4y ago
    The approach I've been using is that I test against a corpus of questions with known answers. Using sample-and-vote, that is asking for some number of variable completions with a non-zero temperature and treating the explicitly computed solutions as votes, has smoothed over issues with basic syntax and logic errors. This is pretty similar to the approach used in the Toolformer paper, other than sample-and-vote, which I believe is novel.
    - triyambakam 4y ago
      In this context what does temperature mean?
      - williamcotton 4y ago
        Temperature is an option passed into the LLM. A temperature of 0 means the response is deterministic, that is that it always returns the same completion. A temperature of greater than zero results in increasing variation in the completion.

williamcotton 4y ago

I've been writing prompts with the intent of producing JSON that contains JS functions to evaluate. Since the response needs to be parsed and evaluated in a specific structure it means that changes need to be made on the evaluation side as well as the prompt template side. So I've been writing the translation examples (few-shot examples [not my favorite term]) as TypeScript, transpiling to JS and compiling with other examples and a prelude, and using the same method to build the translation examples as to build future prompts. It saves a lot of silly mistakes! JavaScript has less tokens than TypeScript, so it seems more economical to convert to JS beforehand instead of passing TS to the LLM! I wouldn't be surprised if TS resulted in better solutions, though... add it to the endless list of things to test out...

BiteCode_dev 4y ago
The problem with that approach is that is very hard to verify that the output is consistent. If you get pure json, you can check it against a schema, and regularly check that chatgpt is not outputting BS (which is does easily). With turing complete code, it's way harder.
- williamcotton 4y ago
  The approach I've been using is that I test against a corpus of questions with known answers. Using sample-and-vote, that is asking for some number of variable completions with a non-zero temperature and treating the explicitly computed solutions as votes, has smoothed over issues with basic syntax and logic errors. This is pretty similar to the approach used in the Toolformer paper, other than sample-and-vote, which I believe is novel.
  - triyambakam 4y ago
    In this context what does temperature mean?
    - williamcotton 4y ago
      Temperature is an option passed into the LLM. A temperature of 0 means the response is deterministic, that is that it always returns the same completion. A temperature of greater than zero results in increasing variation in the completion.