3 ms·

Could you ask an LLM if they were persuaded by a speaker of one side of a debate as a method of evaluation? Ie the bot's before and after opinion based on fine-

by dcsan 3y ago

Could you ask an LLM if they were persuaded by a speaker of one side of a debate as a method of evaluation? Ie the bot's before and after opinion based on fine-tuning with the pro and con arguments? I was also thinking about a society of bots type application where you could have autonomous bot researchers, debaters, judges and audience. Would be interesting to feed in the topics and grab some popcorn

notahacker 3y ago
I think you could ask an LLM if they were persuaded, but I don't think you'd get meaningful data from it Leaving aside the bots that are trained to answer "as an LLM I do not have opinions..." it's going to be a very basic probabilistic yes or no based on tone and numbers of pros and cons rather than knowledge of the surrounding political context and higher order reasoning about the accuracy of the claimed pros and cons.
- rtkwe 3y ago
  Yeah the bot has both no initial position and no internal opinion on fact that an argument could operate on. It might write you a cogent response but it's fundamentally not a question an LLM can meaningfully answer.
Der_Einzige 3y ago
Relevant useful dataset: https://paperswithcode.com/dataset/debatesum https://paperswithcode.com/dataset/debatesum
provability 3y ago
I like this question. I've always wanted to be able to compute, somehow, the "partial derivative of x with respect to y," where x is a proposition and y is an argument or a piece of evidence. It seems to me as though we're closer than ever before. The models are still fairly opaque, but I'm hoping the interpretability research will succeed and allow such introspection!
staticman2 3y ago
If you tell the LLM it is conservative it will report it was persuaded by the conservative speaker, if you tell it it is a liberal it will say it's persuaded by the liberal speaker. It will be happy to roleplay at being persuaded but I don't know why that would be useful?

by dcsan 3y ago

notahacker 3y ago

I think you could ask an LLM if they were persuaded, but I don't think you'd get meaningful data from it Leaving aside the bots that are trained to answer "as an LLM I do not have opinions..." it's going to be a very basic probabilistic yes or no based on tone and numbers of pros and cons rather than knowledge of the surrounding political context and higher order reasoning about the accuracy of the claimed pros and cons.

rtkwe 3y ago
Yeah the bot has both no initial position and no internal opinion on fact that an argument could operate on. It might write you a cogent response but it's fundamentally not a question an LLM can meaningfully answer.

Der_Einzige 3y ago

Relevant useful dataset: https://paperswithcode.com/dataset/debatesum https://paperswithcode.com/dataset/debatesum

provability 3y ago

I like this question. I've always wanted to be able to compute, somehow, the "partial derivative of x with respect to y," where x is a proposition and y is an argument or a piece of evidence. It seems to me as though we're closer than ever before. The models are still fairly opaque, but I'm hoping the interpretability research will succeed and allow such introspection!

staticman2 3y ago

If you tell the LLM it is conservative it will report it was persuaded by the conservative speaker, if you tell it it is a liberal it will say it's persuaded by the liberal speaker. It will be happy to roleplay at being persuaded but I don't know why that would be useful?