5 ms·
Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). For example, their method in the recent pape
by psyklic 3y ago
Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). For example, their method in the recent paper "Language models can explain neurons in language models" [1] is:
1. Using GPT-4, generate a text explanation of a neuron's activations on sample input.
2. Using GPT-4 again, use the text explanation to simulate the neuron on some new text input.
3. Compare the result to the actual neuron's activations on the new text input.
They justify this by saying human contractors do equally poorly at coming up with text descriptions. However, the procedure is such a black box that it is difficult to make scientific conclusions from the results.
[1] https://openai.com/research/language-models-can-explain-neurons-in-language-models https://openai.com/research/language-models-can-explain-neur...
- EGreg 3y agoCan't we just have GPT-4 make the scientific conclusions from these results? /s
- malisper 3y ago> Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). The way OpenAI used GPT-4 is fundamentally different than how GPT-4 was used to score the answers to the MIT exam. In OpenAI's case, they had GPT-4 generate an explanation of when a neuron in GPT-3 would fire. They then gave that explanation back to GPT-4 and had GPT-4 predict when the specific neuron in GPT-3 would fire. The scoring was done by computing the correlation between when GPT-4 predicted the neuron would fire and when it actually fired. The scoring was not done by GPT-4 as was done for the MIT exam In addition OpenAI did have human evaluators score the explanations as well to make sure they were human interpretable[0] [0] https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html#sec-human-scoring https://openaipublic.blob.core.windows.net/neuron-explainer/...
- tehsauce 3y agoCorrect except they did this for gpt2 not gpt3
- psyklic 3y agoIndeed, the OpenAI paper is more well-founded since the "ground truth" was not generated by GPT-4. However, both papers rely on black boxes instead of well-understood procedures. This places the papers on weaker scientific footing. For example, a poor explanation/simulation of a neuron's behavior may simply be a consequence of GPT-4. Instead, a scientist would want to "prove" some form of unexplainability. To do this, a researcher would not use a human at all to explain neuronal behavior. Instead, a simple repeatable algorithm such as topic modeling would be applied. This would lead to significantly stronger scientific conclusions about the neurons. It also proves it is not possible to explain the neuron in some specific sense. An interesting follow-up to the OpenAI paper might be to quantify how much "more powerful" its textual descriptions are than simpler, well-understood techniques such as topic modeling. That could at least reinforce its use.
- behnamoh 3y agoI mean that’s literally the purpose of that paper: to show the capabilities of a blackbox (gpt4).
- sebzim4500 3y agoI don't understand your objection. Step (3) is the one that actually assesses how well the proposed description works, and that is a comparison with the 'real' ground truth.