I think your intuition about this all is right. Maybe we’re holding it wrong, maybe it’s plateaued and won’t get better, or maybe it will get massively better.

I think your intuition about this all is right. Maybe we’re holding it wrong, maybe it’s plateaued and won’t get better, or maybe it will get massively better. Whatever happens, I think it’s the right call to hold a sober opinion— LLMs are just another dumb, expendable tool. If a tool does not consistently produce results, you HAVE to take that at face value. You can’t just remove the numbers bringing down the average and say you have reached 95% success. When you see polar opposite experiences from so many people, the only reasonable takeaway is “so it’s unpredictable, very hard to use, or both.”