This is a real roller coaster of an update. > [S]ome expert testers had indicated that the model behavior “felt” slightly off. > In the end, we decided to lau

This is a real roller coaster of an update. > [S]ome expert testers had indicated that the model behavior “felt” slightly off. > In the end, we decided to launch the model due to the positive signals from the [end-]users who tried out the model. > Looking back, the qualitative assessments [from experts] were hinting at something important Leslie called. He wants to know if you read his paper yet? > Even if these issues aren’t perfectly quantifiable today, All right, I guess not then ... > What we’re learning > Value spot checks and interactive testing more: We take to heart the lesson that spot checks and interactive testing should be valued more in final decision-making before making a model available to any of our users. This has always been true for red teaming and high-level safety checks. We’re learning from this experience that it’s equally true for qualities like model behavior and consistency, because so many people now depend on our models to help in their daily lives. > We need to be critical of metrics that conflict with qualitative testing: Quantitative signals matter, but so do the hard-to-measure ones, and we’re working to expand what we evaluate. Oh, well, some of you get it. At least ... I hope you do.