From the GPT-4 paper (pp20): 2.13 Overreliance ...despite GPT-4’s capabilities, it maintains a tendency to make up facts, to double-down on incorrect informat

From the GPT-4 paper (pp20): 2.13 Overreliance ...despite GPT-4’s capabilities, it maintains a tendency to make up facts, to double-down on incorrect information, and to perform tasks incorrectly. Further, it often exhibits these tendencies in ways that are more convincing and believable than earlier GPT models (e.g., due to authoritative tone or to being presented in the context of highly detailed information that is accurate), increasing the risk of overreliance. Overreliance occurs when users excessively trust and depend on the model, potentially leading to unnoticed mistakes and inadequate oversight. This can happen in various ways: users may not be vigilant for errors due to trust in the model; they may fail to provide appropriate oversight based on the use case and context; or they may utilize the model in domains where they lack expertise, making it difficult to identify mistakes. As users become more comfortable with the system, dependency on the model may hinder the development of new skills or even lead to the loss of important skills. Overreliance is a failure mode that likely increases with model capability and reach. As mistakes become harder for the average human user to detect and general trust in the model grows, users are less likely to challenge or verify the model’s responses. Source: https://arxiv.org/abs/2303.08774 https://arxiv.org/abs/2303.08774 My annotated version: https://lifearchitect.ai/report-card/ https://lifearchitect.ai/report-card/