2 ms·
all ive done with gpt5 for coding was a major db refactor. i had run out of gemini limit for the day. certainly got the job done. I doubt my gpt 20b or ~30b lo
by incomingpain 1y ago
all ive done with gpt5 for coding was a major db refactor. i had run out of gemini limit for the day.
certainly got the job done. I doubt my gpt 20b or ~30b local llm would have been as capable. Overall it was about ~2000 lines of code to change, probably only 100,000 context.
gpt5 didnt one shot it. there were many steps inbetween. At the end, few hours, i had >50 linter warnings from tripled imports, loads of dead code that wouldnt be touched and for some reason gpt5 just couldnt fix any of this. Ended up increasing the warnings and added an error. My expectation is that any of the big guys could immediately fix it. Even restarted fresh context and gpt just wasnt having any of it. im certain even gpt 20b would have completed it in a minute. Curious.
I went to gemini flash, very generic prompt about linter warnings and it fixed it in 30 seconds.
Just kind of weirdness that benchmarks will never be able to catch. It's also going to be very dependent. A rust programmer might have a favourite, whereas python programmer benefits from another model. There can never be a best.
- adinhitlore 1y agoI had similar experience, usually I'd ignore Gemini be it flash or pro but on several occasions it fixed complex errors like it's nothing. Yet when it comes to codegen it is "cheap" on tokens and struggles outputting complex logic. As a great bonus: their easy to setup API is freemium but a generous freemium (google AI studio I mean). My "ecosystem" atm will be something like: gpt5, claude 4.1 - if they both fail: try to fix with gemini. I'd skip Grok for privacy issues mostly not that I completely ignore its capabilities, qwen is good but sometimes 'overengineered' i don't need 400bn , given the large params maybe it will work for non-coding like if you ask it some exotic questions about science: casimir effect, acoustic levitation, ununennium etc etc you name it.
- deleted 1y ago[deleted]