From what I can tell, during the finetuning process OpenAI tried to make chatGPT try as hard as possible to follow instructions, and not point out when the ques

From what I can tell, during the finetuning process OpenAI tried to make chatGPT try as hard as possible to follow instructions, and not point out when the question doesn't make any sense. I'm sure if they had used different criteria during RLHF and used different supervised examples it would have given that answer. It might make the performance worse on other tasks though.