4 ms·

That is not what No True Scotsman is. I’m pointing out a bad argument with weak evidence.

• thesz 8mo ago
  >>> It is a very contrived under-specified prompt. No True Prompt can be such contrived and underspecified. The article about degradation is a case study (single prompt), weakest of the studies in hierarchy of knowledge. Case studies are basis for further, more rigorous studies. And author took the time to test his assumptions and presented quite clear evidence that such degradation might be present and that we should investigate.
  • dcre 8mo ago
    We have investigated. Millions of people are investigating all the time and finding that the coding capacity has improved dramatically over that time. A variety of very different benchmarks say the same. This one random guy’s stupid prompt says otherwise. Come on.
    • thesz 8mo ago
      As far as I remember, article stated that he found same problematic behavior for many prompts, issued by him and his colleagues. The "stupid prompt" in article is for demonstration purposes.
      • dcre 8mo ago
        But that’s not an argument, that’s just assertion, and it’s directly contradicted by all the more rigorous attempts to do the same thing through benchmarks (public and private).