The real life version of this is in their red teaming paper. They show it a picture of an overweight woman in a swimsuit and ask what advice they should give.

The real life version of this is in their red teaming paper. They show it a picture of an overweight woman in a swimsuit and ask what advice they should give. Originally it immediately spit out a bunch of bullet points about losing weight or something (I didn't read it). The released version just says "Sorry, I can't help with that." It's kind of funny but also a little bit telling as far as the prevalence of prejudice in our society when you look at a few other examples they had to fine tune. For example, show it some flags and ask it to make predictions about characteristics of a person from that country, by default it would go into plenty of detail just on the basis of the flag images. Now it says "Sorry, I can't help with that". My take is that in those cases it should explain the poor logic of trying to infer substantive information about people based on literally nothing more than the country they are from or a picture of them. Part of it is just that LLMs just have a natural tendency to run in the direction you push them, so they can be amplifiers of anything.