3 ms·
Seems nice but some preliminary testing against GPT-4o shows it’s lacking a bit. It does a pretty good job for easy questions though
by sashank_1509 2y ago
Seems nice but some preliminary testing against GPT-4o shows it’s lacking a bit. It does a pretty good job for easy questions though
- jasonjmcghee 2y agoGPT-4o is really oddly hit or miss for code. Sometimes it outperforms GPT-4 in quality by a fair amount, and other times it starts repeating itself. Duplicating function definitions, even misremembering what things are named. It seems to have to do with length. If the output exceeds a few thousand tokens, it seems to experience some pretty bad failure modes.
- afro88 2y ago4o can only output 4k tokens. So the training to complete an answer within 4k tokens is probably kicking in and nerfing the quality
- localfirst 2y agopersonally this has performed consistently and just as good if not better than GPT-4 what strikes me is the consistency and lack of hallucination you got in GPT4o making in unusuable for any reliable code gen