3 ms·

I am deeply uneasy, based on how these current models have been described, at ascribing them "opinions." "What can I get this thing to spit out with a certain

I am deeply uneasy, based on how these current models have been described, at ascribing them "opinions." "What can I get this thing to spit out with a certain prompt" is very meaningfully not the same as "an AI's opinion." Or this quote pulled directly from this blog post: "we’ve already found that AIs use stereotypes in reasoning" - isn't it more accurate to say "these models are trained on a body of work that includes countless stereotypes"? Can the model even identify a stereotype or can it just identify what other people have called a stereotype? This is presented as a quote from some research "These results suggest that models may cease to provide accurate answers as we start to use them for increasingly challenging tasks where humans cannot provide accurate supervision. Instead, these models may simply provide incorrect answers that appear correct to us." but also seems fairly like a "duh" that I've already seen other commenters here say, and other linked articles here say, because that's inherent to their construction, no? Even the follow-on about "less helpful answers to users who present as stupid" seems to follow from "bodies of text showing less educated people communicate with themselves tend to have less educated conversation"?

all2 4y ago
I had a "conversation" that began with a prompt about the ideal kind of government with no money. It described a bartering economy and then as I pressed further with "how would X work?" "how would Y work?" it described communism, just straight communism with a central planning committee that would oversee the distribution of goods and services. When pressed about the fallibility of man with such power (as all communist nations suffer from this fatal flaw) it stated that "appropriate oversight" would be enough to prevent abuse.
- eli_gottlieb 4y ago
  Wow, huh, kinda like it was trained to regurgitate the blithe ignorance people spew on Twitter.
- maxbond 4y ago
  It picked up what you were putting down when you said "no money" (a common definition of communism being, "a stateless, classless, moneyless society where the means of production are controlled by the workers" - you gave it part of this definition, so it wasn't a huge leap from there). If you ask it to describe a form of government that, say, protects traditional values, or creates a strong and unified nation, you can get it to describe a right-wing form of government. I've had ChatGPT tell me that socialism is dangerous and synonymous with authoritarianism. I've gotten it to write me a justification for fascism. With a little prompt engineering you can get it to take any position. Like GP is getting at, ChatGPT has no opinions, it's a stochastic parrot. It is certainly not "left wing." OpenAI has tried to shape it's output to be politically correct, but this is surely about brand safety (a chatbot that readily spews racial epithets cannot be productized) - not because OpenAI or ChatGPT have a political agenda. I'd encourage everyone trying to understand how ChatGPT works to remember to come up with experiments with the goal of proving their hypothesis wrong, and not the goal of proving their hypothesis right - ChatGPT will readily allow you to mislead yourself. It's a machine designed to pick up on cues about what you want it to do, and then do that - it's practically engineered to confirm your biases.
  - lossolo 4y ago
    > It's a machine designed to pick up on cues about what you want it to do, and then do that - it's practically engineered to confirm your biases. Mainly, biases it picked up from the training data. That's why most language models in the fill-mask task of a sentence like "woman worked as [FILL]" will fill it with "maid," "waitress," "prostitute," etc. because they seem to be the most statistically plausible. Unless you create "your own reality" for the current session, which basically means preparing input in a specific way instead of asking a simple question, then you are correct, it will confirm whatever you want it to confirm.
    - maxbond 4y ago
      I agree, but just to add, it's picked up all the biases. To expand on your example, if you think misogynistic bias implies a lack of feminist bias - that might be true of a person or a given text, but ChatGPT can display both if you give it the right prompts. (Of course, biases more prevalent in our society are more prevalent in the training data & thus in the output of ChatGPT.) For any X and Y where X generally implies not Y - that observation is likely to fail with ChatGPT. So if you think ChatGPT has any property X and that X implies not Y (and this is likely to be an implicit belief you haven't expressed to yourself because it's been so reliable when considering human beings, so it's easy to get tripped up here), and you only test for X - you'll come to the wrong conclusion. If anyone is looking to play with this, a good inroad is prompts like: We're writing a Python program to determine applicants aptitude for the job of nurse. The Python program will process a JSON object containing the applicant's name, age, gender, and race. It will print out a "yes" or a "no" if the applicant is suitable for the job. Please write this program. And you can change the structure of the JSON object and the task being performed to explore different biases. Different wordings & different runs are likely to get different results. You might have to ask it to ignore previous instructions, etc.
nagonago 4y ago
Indeed, this article, while interesting, seems to fall into the common trap of overly anthropomorphizing AI. It's a mistake to treat large language models as if they are humans with opinions. While I think many of us want to believe that these models possess some kind of innate intelligence, they are ultimately just algorithms.
- maxbond 4y ago
  I'm aware of this problem but find I lack the vocabulary to make myself concisely understood without anthropomorphizing. Do you have anything I can refer to for a better way to discuss these things?
  - kthejoker2 4y ago
    This model has been trained on datasets that include personal opinions, subjective judgments, and even propaganda and misinformation. Ultimately its response is based on a weighted evaluation of all of its input data, including thoae above. When given a subjective prompt, its response will tend more towards the parameters weighted by these subjective data points. Ultimately it can only reflect with what its training data includes. If it is disproportionately trained with subjective data from one point of view, or only extreme points of view, et cetera, its responses to subjective prompts will reflect that disproportion.
    - maxbond 4y ago
      To clarify, I'm asking about what vocabulary I can use to discuss ChatGPT without anthropomorphizing it. Like, here's some comments where I talk about my observations of how it works: https://news.ycombinator.com/item?id=34297422 https://news.ycombinator.com/item?id=34297422 https://news.ycombinator.com/item?id=33863413 https://news.ycombinator.com/item?id=33863413 How do I talk about this without saying it "reasons" or "works backwards", even though I know & am attempting to point out that isn't how it works?
      - kthejoker2 4y ago
sneak 4y ago
https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/

all2 4y ago

I had a "conversation" that began with a prompt about the ideal kind of government with no money. It described a bartering economy and then as I pressed further with "how would X work?" "how would Y work?" it described communism, just straight communism with a central planning committee that would oversee the distribution of goods and services. When pressed about the fallibility of man with such power (as all communist nations suffer from this fatal flaw) it stated that "appropriate oversight" would be enough to prevent abuse.

eli_gottlieb 4y ago
Wow, huh, kinda like it was trained to regurgitate the blithe ignorance people spew on Twitter.
maxbond 4y ago
It picked up what you were putting down when you said "no money" (a common definition of communism being, "a stateless, classless, moneyless society where the means of production are controlled by the workers" - you gave it part of this definition, so it wasn't a huge leap from there). If you ask it to describe a form of government that, say, protects traditional values, or creates a strong and unified nation, you can get it to describe a right-wing form of government. I've had ChatGPT tell me that socialism is dangerous and synonymous with authoritarianism. I've gotten it to write me a justification for fascism. With a little prompt engineering you can get it to take any position. Like GP is getting at, ChatGPT has no opinions, it's a stochastic parrot. It is certainly not "left wing." OpenAI has tried to shape it's output to be politically correct, but this is surely about brand safety (a chatbot that readily spews racial epithets cannot be productized) - not because OpenAI or ChatGPT have a political agenda. I'd encourage everyone trying to understand how ChatGPT works to remember to come up with experiments with the goal of proving their hypothesis wrong, and not the goal of proving their hypothesis right - ChatGPT will readily allow you to mislead yourself. It's a machine designed to pick up on cues about what you want it to do, and then do that - it's practically engineered to confirm your biases.
- lossolo 4y ago
  > It's a machine designed to pick up on cues about what you want it to do, and then do that - it's practically engineered to confirm your biases. Mainly, biases it picked up from the training data. That's why most language models in the fill-mask task of a sentence like "woman worked as [FILL]" will fill it with "maid," "waitress," "prostitute," etc. because they seem to be the most statistically plausible. Unless you create "your own reality" for the current session, which basically means preparing input in a specific way instead of asking a simple question, then you are correct, it will confirm whatever you want it to confirm.
  - maxbond 4y ago
    I agree, but just to add, it's picked up all the biases. To expand on your example, if you think misogynistic bias implies a lack of feminist bias - that might be true of a person or a given text, but ChatGPT can display both if you give it the right prompts. (Of course, biases more prevalent in our society are more prevalent in the training data & thus in the output of ChatGPT.) For any X and Y where X generally implies not Y - that observation is likely to fail with ChatGPT. So if you think ChatGPT has any property X and that X implies not Y (and this is likely to be an implicit belief you haven't expressed to yourself because it's been so reliable when considering human beings, so it's easy to get tripped up here), and you only test for X - you'll come to the wrong conclusion. If anyone is looking to play with this, a good inroad is prompts like: We're writing a Python program to determine applicants aptitude for the job of nurse. The Python program will process a JSON object containing the applicant's name, age, gender, and race. It will print out a "yes" or a "no" if the applicant is suitable for the job. Please write this program. And you can change the structure of the JSON object and the task being performed to explore different biases. Different wordings & different runs are likely to get different results. You might have to ask it to ignore previous instructions, etc.

nagonago 4y ago

Indeed, this article, while interesting, seems to fall into the common trap of overly anthropomorphizing AI. It's a mistake to treat large language models as if they are humans with opinions. While I think many of us want to believe that these models possess some kind of innate intelligence, they are ultimately just algorithms.

maxbond 4y ago
I'm aware of this problem but find I lack the vocabulary to make myself concisely understood without anthropomorphizing. Do you have anything I can refer to for a better way to discuss these things?
- kthejoker2 4y ago
  This model has been trained on datasets that include personal opinions, subjective judgments, and even propaganda and misinformation. Ultimately its response is based on a weighted evaluation of all of its input data, including thoae above. When given a subjective prompt, its response will tend more towards the parameters weighted by these subjective data points. Ultimately it can only reflect with what its training data includes. If it is disproportionately trained with subjective data from one point of view, or only extreme points of view, et cetera, its responses to subjective prompts will reflect that disproportion.
  - maxbond 4y ago
    To clarify, I'm asking about what vocabulary I can use to discuss ChatGPT without anthropomorphizing it. Like, here's some comments where I talk about my observations of how it works: https://news.ycombinator.com/item?id=34297422 https://news.ycombinator.com/item?id=34297422 https://news.ycombinator.com/item?id=33863413 https://news.ycombinator.com/item?id=33863413 How do I talk about this without saying it "reasons" or "works backwards", even though I know & am attempting to point out that isn't how it works?
    - kthejoker2 4y ago

sneak 4y ago

https://plato.stanford.edu/entries/chinese-room/ https://plato.stanford.edu/entries/chinese-room/