Agreed on the lack of value for 13.86% correctness — I noticed that too. This reminds me a little of last year's hype around AutoGPT et al (at around the same t

Agreed on the lack of value for 13.86% correctness — I noticed that too. This reminds me a little of last year's hype around AutoGPT et al (at around the same time of year, oddly enough); it's very promising as a measure of how far we've come since just a few years ago when that metric would be 0%, but it doesn't seem super usable at the moment. That being said, something is definitely coming. 50% correctness would probably be well worth using — simple copy/paste between my editor and GPT4 has been useful for me, and that's much less likely to completely solve an issue in one shot — and not only will small startups doing finetunes be grinding towards better results... The big labs will be too, and releasing improved foundation models that the startups can then continue finetuning. I don't think a new AI winter is on the horizon yet; Meta has plenty of reason to keep pushing out better stuff, both from a product perspective (glasses) and from an efficiency perspective (internal codegen), and OpenAI doesn't seem particularly at risk of stopping since Microsoft is using them both to batter Google on search (by having more people use ChatGPT for general question answering than using Google search), and to claw marketshare from Amazon in their cloud offerings. Similarly, some AI products have already found product/market fit; Midjourney bootstrapped from 0 to $200MM ARR (!) for example, purely on the basis of monthly subscriptions, by disrupting the stock image industry pretty convincingly.