4 ms·

You could potentially introduce an occasional prompt and check the answer against a few LLMs to see if they match.

adventured 3y ago
Good LLM outputs are typically not the same for every identical query. For now that makes AI checkers fairly incompetent (and leads to disastrous teacher results when trying to find the students using AI). If I ask even just GPT 3.5 something like: "Who was John Adams?" - it'll give me a slightly varied answer pretty much every single time (even if I modify its settings to make it less creative with responses, it'll still usually vary a bit). Here is a simple API hit on gpt-3.5-turbo-16k-0613 Output 1) John Adams was an American statesman, lawyer, diplomat, and Founding Father who served as the second President of the United States from 1797 to 1801. He was one of the key figures in the American Revolution and played a crucial role in drafting the Declaration of Independence. Adams also served as the first Vice President under George Washington. He was known for his strong advocacy of republicanism and his belief in a strong central government. Adams was a prolific writer and his letters and writings provide valuable insights into the early years of the United States. Output 2) John Adams was an American statesman, lawyer, diplomat, and Founding Father who served as the second President of the United States from 1797 to 1801. He was one of the key figures in the American Revolution and played a crucial role in drafting the Declaration of Independence. Adams was also a strong advocate for the separation of powers and a strong central government. He was known for his intellect and commitment to public service. And so on.
- samstave 3y ago
  What if every output from a prompt was a QR code which led the user to a page with the actual prompt and the output and this is what AI was relegated to do in order to ad some modicum of proof-of-prompt-custodianship... so the only way one could see the output of the AI was to read the QR code which resulted in the intput & the output?
  - adventured 3y ago
    LLM content is going to be pervasive, to understate it. This stuff is going everywhere. It won't be relegated. The open source variations in the wild aren't going to necessarily follow any such rules. So even if eg a government wanted to lock it down that way, the mass of AI content spilling over the walls anyway would void the effort. I doubt the genie is going back into the bottle or can be easily controlled. We'll come up with some basic ways to discern AI content, one of which will be a government regulation (or standards body) stipulating all AI content must/should be labeled as such (with movie like rating gradiants: entirely AI (E-AI), partially AI (P-AI), no AI (N-AI); or something like that). If we're lucky it'll just be a content industry standard that will be developed, without forced government labeling. QR codes or the equivalent could play a role there (QR code next to the content label; end-point data includes where it came from, when), although someone has to host a permanent repository for the end-point data.
  - orf 3y ago
    … then people would just copy-paste the output after scanning the QR code? Like, what??
flangola7 3y ago
How's would you check that? What if they're using a self hosted LLM?
- IshKebab 3y ago
  You think mturk workers have the resources and expertise needed to set up their own GPT?
  - flangola7 3y ago
    That doesn't address my question. I didn't ask anything about their resources or expertise.

Good LLM outputs are typically not the same for every identical query. For now that makes AI checkers fairly incompetent (and leads to disastrous teacher results when trying to find the students using AI). If I ask even just GPT 3.5 something like: "Who was John Adams?" - it'll give me a slightly varied answer pretty much every single time (even if I modify its settings to make it less creative with responses, it'll still usually vary a bit). Here is a simple API hit on gpt-3.5-turbo-16k-0613 Output 1) John Adams was an American statesman, lawyer, diplomat, and Founding Father who served as the second President of the United States from 1797 to 1801. He was one of the key figures in the American Revolution and played a crucial role in drafting the Declaration of Independence. Adams also served as the first Vice President under George Washington. He was known for his strong advocacy of republicanism and his belief in a strong central government. Adams was a prolific writer and his letters and writings provide valuable insights into the early years of the United States. Output 2) John Adams was an American statesman, lawyer, diplomat, and Founding Father who served as the second President of the United States from 1797 to 1801. He was one of the key figures in the American Revolution and played a crucial role in drafting the Declaration of Independence. Adams was also a strong advocate for the separation of powers and a strong central government. He was known for his intellect and commitment to public service. And so on.

samstave 3y ago
What if every output from a prompt was a QR code which led the user to a page with the actual prompt and the output and this is what AI was relegated to do in order to ad some modicum of proof-of-prompt-custodianship... so the only way one could see the output of the AI was to read the QR code which resulted in the intput & the output?
- adventured 3y ago
  LLM content is going to be pervasive, to understate it. This stuff is going everywhere. It won't be relegated. The open source variations in the wild aren't going to necessarily follow any such rules. So even if eg a government wanted to lock it down that way, the mass of AI content spilling over the walls anyway would void the effort. I doubt the genie is going back into the bottle or can be easily controlled. We'll come up with some basic ways to discern AI content, one of which will be a government regulation (or standards body) stipulating all AI content must/should be labeled as such (with movie like rating gradiants: entirely AI (E-AI), partially AI (P-AI), no AI (N-AI); or something like that). If we're lucky it'll just be a content industry standard that will be developed, without forced government labeling. QR codes or the equivalent could play a role there (QR code next to the content label; end-point data includes where it came from, when), although someone has to host a permanent repository for the end-point data.
- orf 3y ago
  … then people would just copy-paste the output after scanning the QR code? Like, what??

flangola7 3y ago

How's would you check that? What if they're using a self hosted LLM?

IshKebab 3y ago
You think mturk workers have the resources and expertise needed to set up their own GPT?
- flangola7 3y ago
  That doesn't address my question. I didn't ask anything about their resources or expertise.