I think this is a case where one really can't be sure what's going on unless you work at OpenAI. OpenAI claims that they aren't hand-coding outputs for specific

I think this is a case where one really can't be sure what's going on unless you work at OpenAI. OpenAI claims that they aren't hand-coding outputs for specific inputs, and I think that's probably correct, because it seems impractical. But I think the evidence in these posts makes it seem pretty likely that their staff is doing *something* active. I think the most likely possibility is that they do have team members who look at perhaps some sample of conversations where they think it's likely that GPT-3 is not handling some input well, and use those as test cases for their team to debug and improve the model. If it's the case that the OpenAPI team working on GPT-3 has a way to make tactical adjustments (perhaps be extending their training data set in a way that their engineers expect to fix the bug), then I think them having a team that squashes bugs, especially those reported publicly, seems like a plausible explanation for bugs being fixed suspiciously soon after a user finds a good one. In a normal software product designed to solve a problem that is imprecise/heuristic in nature (and with a privacy disclaimer/model that explicitly permits this sort of development), that'd be expected and great, and the only question would be whether one can make the problem-specific bug squashing work easy to scale (i.e. you don't want it to require a bunch of time from a machine learning expert to fix each bug). For many applications where one can imagine using something like GPT-3 as an input, it's not really a problem to have an error rate that can be reduced through investment as long as one can figure out a way to quantify those things and your application is one where an acceptable error rate is possible. For GPT-3 in particular, they might not want to confirm this class of explanation, because they're marketing the technology as a little big magic.