3 ms·
> If there is indeed no degradation how could the perceived degradation be explained? By being disproportionately impressed previously. Maybe in the early days
by latexr 2y ago
> If there is indeed no degradation how could the perceived degradation be explained?
By being disproportionately impressed previously. Maybe in the early days people were so impressed by their little play experiments they forgave the shortcomings. Now that the novelty is wearing off and they try to use it for productive work, the scales tipped and failures are given more weight.
- xbmcuser 2y agoWhen chat gpt first came out I was able to feed it some text to parse and then create python scripts to process similar texts and create csv and excel files from those. I was able to create a basic working python scripts in 1-2 hours. And very complex scripts over a couple of days. I recently tried to do the same with chatgpt again and simply am unable to. I wish I had saved the exact text I fed into chat gpt then so that I could see if it was just my question was wrong or free version of chatgpt has now been hobbled
- PoignardAzur 2y agoChatGPT has a query history. Can't you fish the first queries you made on your account?
- xbmcuser 2y agoLike I said I wish I kept it this was almost 2 years ago I have deleted my query history many times since then
- eltoxo 2y agoI don't know, my experience is all this is really hard to judge. I would have a really hard time saying that 4 in April 2023 was better than 4o now though. I have always wonder if it matters what time of day you are using it too. I feel like 4am EST works better than 4PM EST but it is so hard to judge. I think there is so much difference too with just how the prompt is phrased so it ends up feeling like some days it is is good and some days it sucks. That is coupled with I have got a bad result before, opened a new chat window, pasted the exact same prompt and got a good result. If I had to bet, I imagine it is like flipping quarters. Sometimes you will get runs of heads, sometimes runs of tails and sometimes a real mixed bag of both.
- PeterStuer 2y agoNo, you are wrong, and anyone that was an early adopter will tell you. At the start there were many Python coding tasks it completed flawlessly one shot, that all later iterations needed 5 takes for if they managed to get it at all. And that was using the early 3 series ChatGPT, which they rapidly nerfed into the ground.While the current GPT-4 model feels better in some other less quantifiable ways, Python coding hasn't recovered.
- whbrown 2y agoThere are even some benchmarks which have caught 'lazy coding' regressions with their latest models[1]. I recall my best experience with their models was last year with an early version of the advanced data analysis feature where it would write a script, write tests for it, run the tests, update the code and/or tests, and re-run them. Presumably that was too expensive, and now it feels like pulling teeth to get the same result. [1] https://aider.chat/2024/04/09/gpt-4-turbo.html https://aider.chat/2024/04/09/gpt-4-turbo.html
- omega3 2y agoI've seen different type of complaint most frequently which doesn't match the pattern you've given: t0: novel task 'foo' is successful t1: novel task 'bar' (similar to 'foo') fails Common suggestion here is to execute 'foo' again but the issue here is that the response to 'foo' at t > 0 might just be cached version of response at t0. I think a good analogy here would be an interaction with a human that suffers from cognitive decline. You might get correct answer everyday to the same question but with novel question you notice the degradation in the quality of responses.
- bastawhiz 2y agoI'll anecdotally say that Copilot doesn't give me solutions that work nearly as well lately. It used to be the case that I could have it generate whole classes by prompting only the method signatures. Now it can hardly generate a single function without needing fixes for things like hallucinated variables (in its own code!) or invalid syntax. Even just getting it to output the correct number of closing braces/parens has gotten noticeably worse.
- Spooky23 2y agoThey tweak it to make it safer. Every time that happens, it gets a little dumber.
- PeterStuer 2y agoI think in this case it is more tweaking for inference cost. The full models are never exposed as using them at scale is too expensive.