4 ms·

I've been trying something similar with parliamentary debates. They're long winded, often full of empty speech, and a chore to read. The LLMs are able to hone

I've been trying something similar with parliamentary debates. They're long winded, often full of empty speech, and a chore to read. The LLMs are able to hone in on the details and provide interesting responses like "What questions were asked of the minister that they failed to address" and "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer" Crucially, they're well transcribed: https://api.oireachtas.ie/ https://api.oireachtas.ie/

nnechm 3y ago
I think one thing you can try is to figure out what lies where, so chunking arbitrarily will not work as well as chunking with headings for e.g For e.g, if the question is: "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer." An embedding based search will find it fairly difficult to match this against a text. Based on my experience, you have to figure out what is a nonanswer first (i don't think that's easy but gpt4 is very good at a lot of language based stuff.) you can try Q1: question A1: Answer then prompt GPT, do you think A1 answers the question and then save it. And then, Q1: question, A1: Answer, Q2: Follow up based on the following questions, do you think A1 answers Q1 and then save it to a db. You can then augment it in the code with our own knowledge of how politicians lie, using certain words etc :) to improve what gpt4 might miss...
- anotherhue 3y ago
  > "We're definitely moving forward with strong measures to improve the situation, and while progress might not be immediately visible, we're fully engaged in this essential journey."
  - nnechm 3y ago
    This is how GPT4 answers the question... I assumed the question, but presumably will be along those lines :) https://chat.openai.com/share/c736abf4-ae3c-4fbd-9427-b7d2f9413213 https://chat.openai.com/share/c736abf4-ae3c-4fbd-9427-b7d2f9...
  - hef19898 3y ago
    Man, it seems the people writing politicians, or managers, statements seem at serious risk being replaced with a LLM...
dcsan 3y ago
Could you ask an LLM if they were persuaded by a speaker of one side of a debate as a method of evaluation? Ie the bot's before and after opinion based on fine-tuning with the pro and con arguments? I was also thinking about a society of bots type application where you could have autonomous bot researchers, debaters, judges and audience. Would be interesting to feed in the topics and grab some popcorn
- notahacker 3y ago
  I think you could ask an LLM if they were persuaded, but I don't think you'd get meaningful data from it Leaving aside the bots that are trained to answer "as an LLM I do not have opinions..." it's going to be a very basic probabilistic yes or no based on tone and numbers of pros and cons rather than knowledge of the surrounding political context and higher order reasoning about the accuracy of the claimed pros and cons.
  - rtkwe 3y ago
    Yeah the bot has both no initial position and no internal opinion on fact that an argument could operate on. It might write you a cogent response but it's fundamentally not a question an LLM can meaningfully answer.
- Der_Einzige 3y ago
  Relevant useful dataset: https://paperswithcode.com/dataset/debatesum https://paperswithcode.com/dataset/debatesum
- provability 3y ago
  I like this question. I've always wanted to be able to compute, somehow, the "partial derivative of x with respect to y," where x is a proposition and y is an argument or a piece of evidence. It seems to me as though we're closer than ever before. The models are still fairly opaque, but I'm hoping the interpretability research will succeed and allow such introspection!
- staticman2 3y ago
  If you tell the LLM it is conservative it will report it was persuaded by the conservative speaker, if you tell it it is a liberal it will say it's persuaded by the liberal speaker. It will be happy to roleplay at being persuaded but I don't know why that would be useful?

nnechm 3y ago

I think one thing you can try is to figure out what lies where, so chunking arbitrarily will not work as well as chunking with headings for e.g For e.g, if the question is: "What should the opposition leader have mentioned in their response that the minister would have found difficult to answer." An embedding based search will find it fairly difficult to match this against a text. Based on my experience, you have to figure out what is a nonanswer first (i don't think that's easy but gpt4 is very good at a lot of language based stuff.) you can try Q1: question A1: Answer then prompt GPT, do you think A1 answers the question and then save it. And then, Q1: question, A1: Answer, Q2: Follow up based on the following questions, do you think A1 answers Q1 and then save it to a db. You can then augment it in the code with our own knowledge of how politicians lie, using certain words etc :) to improve what gpt4 might miss...

anotherhue 3y ago
> "We're definitely moving forward with strong measures to improve the situation, and while progress might not be immediately visible, we're fully engaged in this essential journey."
- nnechm 3y ago
  This is how GPT4 answers the question... I assumed the question, but presumably will be along those lines :) https://chat.openai.com/share/c736abf4-ae3c-4fbd-9427-b7d2f9413213 https://chat.openai.com/share/c736abf4-ae3c-4fbd-9427-b7d2f9...
- hef19898 3y ago
  Man, it seems the people writing politicians, or managers, statements seem at serious risk being replaced with a LLM...

dcsan 3y ago

Could you ask an LLM if they were persuaded by a speaker of one side of a debate as a method of evaluation? Ie the bot's before and after opinion based on fine-tuning with the pro and con arguments? I was also thinking about a society of bots type application where you could have autonomous bot researchers, debaters, judges and audience. Would be interesting to feed in the topics and grab some popcorn

notahacker 3y ago
I think you could ask an LLM if they were persuaded, but I don't think you'd get meaningful data from it Leaving aside the bots that are trained to answer "as an LLM I do not have opinions..." it's going to be a very basic probabilistic yes or no based on tone and numbers of pros and cons rather than knowledge of the surrounding political context and higher order reasoning about the accuracy of the claimed pros and cons.
- rtkwe 3y ago
  Yeah the bot has both no initial position and no internal opinion on fact that an argument could operate on. It might write you a cogent response but it's fundamentally not a question an LLM can meaningfully answer.
Der_Einzige 3y ago
Relevant useful dataset: https://paperswithcode.com/dataset/debatesum https://paperswithcode.com/dataset/debatesum
provability 3y ago
I like this question. I've always wanted to be able to compute, somehow, the "partial derivative of x with respect to y," where x is a proposition and y is an argument or a piece of evidence. It seems to me as though we're closer than ever before. The models are still fairly opaque, but I'm hoping the interpretability research will succeed and allow such introspection!
staticman2 3y ago
If you tell the LLM it is conservative it will report it was persuaded by the conservative speaker, if you tell it it is a liberal it will say it's persuaded by the liberal speaker. It will be happy to roleplay at being persuaded but I don't know why that would be useful?