41 ms·
so I have done a lot of transcripts, coding, one versus the other (gpt4 vs ultra). Often simple prompts like refactor this code or convert this python to types
by DannyBee 3y ago
so I have done a lot of transcripts, coding, one versus the other (gpt4 vs ultra). Often simple prompts like refactor this code or convert this python to typescript.
My experience is that Gemini ultra understands the code better, but doesn’t always give me as complete of results (they seem to limit output length more)
Beyond that it is very smart. I’ve had to tell me this code packs 12 bit integers into different parts of an array using the following encodinv. Which most people would not figure out from the code as written. It then will say you can actually do that with this neat little translate function that you never knew about.
It will then get the code very slightly wrong. if I tell it not to use the cool function, it will actually get the code, right.
GPT4 has no idea what the code is doing but can clean it up a bit.
so it’s like ultra is too clever by half sometimes.
That said, I have fed thousands of lines of code into both of them and asked them to refactor it, and neither one of them made more than one error. All code otherwise compiled and worked first try.
this is code that can’t possibly be in their training sets, it’s basically handwritten python that was written, based on an old x86 binary that nobody has the source to anymore. so the code is basically garbage, and what it is doing doesn't, say, appear on GitHub in a nicer form.
Both gpt4 and Gemini ultra were able to make the code, look like clean, idiomatic, python, or typescript without any work on my part. except for the one bug each. which, for 8000 to 10,000 lines of code is not bad.
The GPT4 inserted bug was more insidious. It changed (the equivalent of) (uint8)'a' to (uint8)'a' - '0' for no reason when converting some code to typescript. Not sure if that is representative of anything
if I do the same with any of the other “top” models ( from can ai code, etc), most of them can’t even generate correct working code for all the input, let alone good code. Most aren't even close.
- jstummbillig 3y ago> That said, I have fed thousands of lines of code into both of them and asked them to refactor it, and neither one of them made more than one error. All code otherwise compiled and worked first try. I would be very interested to get a more detailed scope of what you did here. Feeding thousands of lines of code into GPT4 and getting a near perfect refactor does very much NOT sound like my experience, but it seems highly desirable.
- deleted 3y ago[deleted]
- maxwelljoslyn 3y agoSeconded. I am also keenly interested in learning more. It would be a great boon on my current project to be able to let the AI refactor mountains of legacy code (while taking steps to ensure the results are trustworthy and equivalent, of course.)
- maaanu 3y agoYes, I've observed the same phenomenon. The more detailed my prompts are, the more errors GPT tends to make. I use it as a partner to discuss implementation ideas, before I start coding. That works very well, because gpt and I usually find somethings, that I missed at a first glance. But coding with gpt or co-pilot is too disrupted for me.
- q7xvh97o2pDhNrh 3y ago> The more detailed my prompts are, the more errors GPT tends to make. I use it as a partner to discuss implementation ideas, before I start coding. This is interesting; I do a third thing that's the exact opposite of both of you. I generally know the architecture I want, so I just slowly spoon-feed it to GPT-4 and have it build each of the components with the right interfaces to fit together. Usually, building each component requires a few rounds of hand-holding it through various refactoring. Its first draft of code tends to be, for lack of a better description, totally lacking in any creativity or elegance --- generally bland, insipid, and mundane stuff. But it'll gladly take refactoring directions to get it into shape, so it's still very quick to get a whole bunch of components written up. Then I just glue everything together on my own. And the whole experience is still way better than writing a bunch of boring boilerplate code by hand.
- DannyBee 3y agoI'm happy to share transcripts if you email me. I'm not sure what you are feeding it. My scope is closer to a file at a time of fairly mostly self-contained python or C and asking it to clean it up or convert it to typescript. I can imagine lots of scenarios it doesn't work well. In mine, it does, and I have plenty of transcripts showing that :)
- emmelaich 3y ago> what it is doing doesn't, say, appear on GitHub in a nicer form Surely many similar bits of code appear? I'd have to see your problem though. I'm thinking of ETL type code which has to mangle various custom exotic encodings into others and vice-versa.
- marktl 3y agoWith GPT 4's context window I've had no luck with more than 2k lines of code in a single prompt.