4 ms·

Honestly, it’s kind of difficult to believe. GPT4 is great at this stuff, but iterative refinement doesn’t work in my experience. As the conversation increase

Honestly, it’s kind of difficult to believe. GPT4 is great at this stuff, but iterative refinement doesn’t work in my experience. As the conversation increases, the previous context is lost and the generated code deviates from its previous behaviour. For example, “fix this bug…” can easily result in a solution that breaks some other thing. You can also see code generated in thread (1) that does exist in the final result (2), suggesting that (since this is the very top of the code), they were getting chatGPT to iteratively generate 600+ line segments. I severely doubt this. Creating a new Slingshot on line 20 after it is defined on line 500? That is extraordinarily unlikely unless you specifically prompted it to do that. “loadImage('stone2.png');”, it just happened to pick the right file names? The right sprite sizes? You provided all that in a prompt and it wrote the code? Come onnnn… show us the actual prompt you used. It seems much more likely they generated a set of class objects relatively independently, then manually assembled them into a larger file, copied the entire thing as input and then crafted a “code prompt” like “write a function that does such and such”. It’s not impossible they used prompts like they claim (3), but I feel they are (for likes and cred) vastly overstating the “it did all the coding” part of this project. I feel they probably hand wrote some of the code (or assembled it) and used it as input + a “now do this also” style prompt, so the output was “100% generated”, but not in the way people are assuming. This approach tends to make GPT4 rewrite the existing code, but unless you specifically ask for (or add) comments describing the intent through out the code (missing in most of the generated code), it will drift from the previous functionality. With no test suite to verify, you won’t notice this subtle drift and things just break. There’s no mention of either of these things being done by the author. Further more, this user has a vested interest (4) in selling training materials for AI, so it’s in their interest to appear to be an expert at this, and has provided (even when asked on X) no additional details, no “step by step” git repo with history, no actual prompts they’ve used. Given the lack of details and the frankly unbelievable results, I think there’s fair call to be sceptical in this case. You could generate this kind of thing from models such as codellama 34B, or GPT 3.5; but not using the method as described. I’m… not convinced you could do it with gpt4. The prompts seem too stupid to be real (5)… but I happy to be proved wrong with more details. GPT4 is good. [1] - https://nitter.net/pic/orig/media%2FF9xoI8mXgAAn7v9.jpg https://nitter.net/pic/orig/media%2FF9xoI8mXgAAn7v9.jpg [2] - https://bestaiprompts.art/angry-pumpkins/sketch.js https://bestaiprompts.art/angry-pumpkins/sketch.js [3] - https://nitter.net/javilopen/status/1719363669685916095#m https://nitter.net/javilopen/status/1719363669685916095#m [4] - https://javilopen.substack.com/ https://javilopen.substack.com/ [5] - “Now, make the monsters circular, and be very careful: apply the same technique that already exists for the rectangular ones regarding scaling and collision area, and don't mess it up like before. ”

inciampati 3y ago
Context is 8k and it's quadratic. It "sees" everything in that window. If you want to have a long conversation try Claude or some of the 32k models. Claude uses a strange kind of attention that isn't always as precise but it's very good at finding key information in huge documents.
- wokwokwok 3y ago
  Key information is not relevant for generating code. You have to generate specific api calls with specific semantics not “high level summary” and “key information” . You have to generate token sequences that are functionally equivalent to an exact segment of the input prompt. A lower precision is less useful for code generation. How do you maintain existing code functionality when you give code + “refactor for x” as an input? I’m skeptical you’ve tried this and know what you’re talking about.
  - inciampati 3y ago
    You are welcome to be skeptical and I will happily continue to use these tools for drafting code and writing every day. I usually provide documentation for the systems I want to use on input. Then I include source code of anything I'm hacking on. If none exists I build it up using "let's do this step by step" where the first step is outlining or architecture and subsequent steps fill out components. The same pattern works for long form prose. Having 100k tokens is almost enough. It means the model can attend to a ton of relevant information. Also, this works well when the system gives you patches. I wouldn't say "rewrite this entire thing to make this one change" because if it's anything non trivial it will shift. Again I see the same patterns for code and text. Ask for small changes, contained updates. Guide the model to work in small increments, using abstraction to deal with high level broad topics.
    - wokwokwok 3y ago
      Seriously? You wouldn’t use it refactor because the implementation will drift, but you’d use it to generate a patch to apply to the code? Hm. Well, I don’t think that style is relevant or practical for large scale code generation as shown even in this ~600 line example. …but, I’d love to see the details of how it could done, end to end, by someone who has done something like that successfully and is willing to prove it by sharing the prompts and outputs.

inciampati 3y ago

Context is 8k and it's quadratic. It "sees" everything in that window. If you want to have a long conversation try Claude or some of the 32k models. Claude uses a strange kind of attention that isn't always as precise but it's very good at finding key information in huge documents.

wokwokwok 3y ago
Key information is not relevant for generating code. You have to generate specific api calls with specific semantics not “high level summary” and “key information” . You have to generate token sequences that are functionally equivalent to an exact segment of the input prompt. A lower precision is less useful for code generation. How do you maintain existing code functionality when you give code + “refactor for x” as an input? I’m skeptical you’ve tried this and know what you’re talking about.
- inciampati 3y ago
  You are welcome to be skeptical and I will happily continue to use these tools for drafting code and writing every day. I usually provide documentation for the systems I want to use on input. Then I include source code of anything I'm hacking on. If none exists I build it up using "let's do this step by step" where the first step is outlining or architecture and subsequent steps fill out components. The same pattern works for long form prose. Having 100k tokens is almost enough. It means the model can attend to a ton of relevant information. Also, this works well when the system gives you patches. I wouldn't say "rewrite this entire thing to make this one change" because if it's anything non trivial it will shift. Again I see the same patterns for code and text. Ask for small changes, contained updates. Guide the model to work in small increments, using abstraction to deal with high level broad topics.
  - wokwokwok 3y ago
    Seriously? You wouldn’t use it refactor because the implementation will drift, but you’d use it to generate a patch to apply to the code? Hm. Well, I don’t think that style is relevant or practical for large scale code generation as shown even in this ~600 line example. …but, I’d love to see the details of how it could done, end to end, by someone who has done something like that successfully and is willing to prove it by sharing the prompts and outputs.