4 ms·

> Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). The way OpenAI used GPT-4 is fundamentall

> Even research from OpenAI has attempted to use GPT-4 as quasi-ground truth (as a replacement for human evaluators). The way OpenAI used GPT-4 is fundamentally different than how GPT-4 was used to score the answers to the MIT exam. In OpenAI's case, they had GPT-4 generate an explanation of when a neuron in GPT-3 would fire. They then gave that explanation back to GPT-4 and had GPT-4 predict when the specific neuron in GPT-3 would fire. The scoring was done by computing the correlation between when GPT-4 predicted the neuron would fire and when it actually fired. The scoring was not done by GPT-4 as was done for the MIT exam In addition OpenAI did have human evaluators score the explanations as well to make sure they were human interpretable[0] [0] https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html#sec-human-scoring https://openaipublic.blob.core.windows.net/neuron-explainer/...

tehsauce 3y ago
Correct except they did this for gpt2 not gpt3
psyklic 3y ago
Indeed, the OpenAI paper is more well-founded since the "ground truth" was not generated by GPT-4. However, both papers rely on black boxes instead of well-understood procedures. This places the papers on weaker scientific footing. For example, a poor explanation/simulation of a neuron's behavior may simply be a consequence of GPT-4. Instead, a scientist would want to "prove" some form of unexplainability. To do this, a researcher would not use a human at all to explain neuronal behavior. Instead, a simple repeatable algorithm such as topic modeling would be applied. This would lead to significantly stronger scientific conclusions about the neurons. It also proves it is not possible to explain the neuron in some specific sense. An interesting follow-up to the OpenAI paper might be to quantify how much "more powerful" its textual descriptions are than simpler, well-understood techniques such as topic modeling. That could at least reinforce its use.
- behnamoh 3y ago
  I mean that’s literally the purpose of that paper: to show the capabilities of a blackbox (gpt4).

tehsauce 3y ago

Correct except they did this for gpt2 not gpt3

psyklic 3y ago

Indeed, the OpenAI paper is more well-founded since the "ground truth" was not generated by GPT-4. However, both papers rely on black boxes instead of well-understood procedures. This places the papers on weaker scientific footing. For example, a poor explanation/simulation of a neuron's behavior may simply be a consequence of GPT-4. Instead, a scientist would want to "prove" some form of unexplainability. To do this, a researcher would not use a human at all to explain neuronal behavior. Instead, a simple repeatable algorithm such as topic modeling would be applied. This would lead to significantly stronger scientific conclusions about the neurons. It also proves it is not possible to explain the neuron in some specific sense. An interesting follow-up to the OpenAI paper might be to quantify how much "more powerful" its textual descriptions are than simpler, well-understood techniques such as topic modeling. That could at least reinforce its use.

behnamoh 3y ago
I mean that’s literally the purpose of that paper: to show the capabilities of a blackbox (gpt4).