3 ms·
It seems like a `ranking_system_prompt` is used to rank the output of other prompts, which is pretty cool! > Your job is to rank the quality of two outputs gen
by arbol 3y ago
It seems like a `ranking_system_prompt` is used to rank the output of other prompts, which is pretty cool!
> Your job is to rank the quality of two outputs generated by different prompts. The prompts are used to generate a response for a given task. You will be provided with the task description, the test prompt, and two generations - one for each system prompt. Rank the generations in order of quality. If Generation A is better, respond with 'A'. If Generation B is better, respond with 'B'. Remember, to be considered 'better', a generation must not just be good, it must be noticeably superior to the other. Also, keep in mind that you are a very harsh critic. Only rank a generation as better if it truly impresses you more than the other. Respond with your ranking, and nothing else. Be fair and unbiased in your judgement.
Source: https://github.com/mshumer/gpt-prompt-engineer/blob/main/gpt_prompt_engineer.ipynb https://github.com/mshumer/gpt-prompt-engineer/blob/main/gpt...
- axlee 3y agoHow did they rank the ranking prompt?
- archargelod 3y agoThey prey extra hard for this one, so the almighty Machine God blesses the results and they won't bring down whole thing. /s
- mistymountains 3y agoThey don’t. They simply assume the model’s most likely output is meaningfully correlated with true rankings even though it was never trained on this task and certainly has not been trained to output the most likely prompt given a prompt in some meaningful order. It’s hogwash.
- A-Train 3y agoI'm pretty surprised more people dont use logit biases to call openai with. Checking if something is either a or b means that the tokens for those letters must be 100 weight which means they will be chosen no matter what and no other character is allowed.
- msp26 3y agoIf you're using OAI function calling, you can define a json schema with boolean values. If you have a handful of values, you can use an enum with a list of possible values. Unless your prompt seriously conflicts with the schema, it's pretty consistent.
- fdondi 3y agoWait, so we are asking the innkeeper if the wine is good? If the model thinks say that the current year is 2021 it will rate higher a response that does? There should be at a minimum a way to print which comparisons were made, so that you could double-check if you agree with those. Also, we could modify the prompt to explain why the decision was made.
- toxicFork 3y agoA neat feature would be able to switch between autoranking and manual ranking.