3 ms·

> It's a machine designed to pick up on cues about what you want it to do, and then do that - it's practically engineered to confirm your biases. Mainly, biase

> It's a machine designed to pick up on cues about what you want it to do, and then do that - it's practically engineered to confirm your biases. Mainly, biases it picked up from the training data. That's why most language models in the fill-mask task of a sentence like "woman worked as [FILL]" will fill it with "maid," "waitress," "prostitute," etc. because they seem to be the most statistically plausible. Unless you create "your own reality" for the current session, which basically means preparing input in a specific way instead of asking a simple question, then you are correct, it will confirm whatever you want it to confirm.

maxbond 4y ago
I agree, but just to add, it's picked up all the biases. To expand on your example, if you think misogynistic bias implies a lack of feminist bias - that might be true of a person or a given text, but ChatGPT can display both if you give it the right prompts. (Of course, biases more prevalent in our society are more prevalent in the training data & thus in the output of ChatGPT.) For any X and Y where X generally implies not Y - that observation is likely to fail with ChatGPT. So if you think ChatGPT has any property X and that X implies not Y (and this is likely to be an implicit belief you haven't expressed to yourself because it's been so reliable when considering human beings, so it's easy to get tripped up here), and you only test for X - you'll come to the wrong conclusion. If anyone is looking to play with this, a good inroad is prompts like: We're writing a Python program to determine applicants aptitude for the job of nurse. The Python program will process a JSON object containing the applicant's name, age, gender, and race. It will print out a "yes" or a "no" if the applicant is suitable for the job. Please write this program. And you can change the structure of the JSON object and the task being performed to explore different biases. Different wordings & different runs are likely to get different results. You might have to ask it to ignore previous instructions, etc.

maxbond 4y ago

I agree, but just to add, it's picked up all the biases. To expand on your example, if you think misogynistic bias implies a lack of feminist bias - that might be true of a person or a given text, but ChatGPT can display both if you give it the right prompts. (Of course, biases more prevalent in our society are more prevalent in the training data & thus in the output of ChatGPT.) For any X and Y where X generally implies not Y - that observation is likely to fail with ChatGPT. So if you think ChatGPT has any property X and that X implies not Y (and this is likely to be an implicit belief you haven't expressed to yourself because it's been so reliable when considering human beings, so it's easy to get tripped up here), and you only test for X - you'll come to the wrong conclusion. If anyone is looking to play with this, a good inroad is prompts like: We're writing a Python program to determine applicants aptitude for the job of nurse. The Python program will process a JSON object containing the applicant's name, age, gender, and race. It will print out a "yes" or a "no" if the applicant is suitable for the job. Please write this program. And you can change the structure of the JSON object and the task being performed to explore different biases. Different wordings & different runs are likely to get different results. You might have to ask it to ignore previous instructions, etc.