4 ms·

I tried using them for a game server, written in C, as a test case and I came to the same conclusions. It did pretty well at making something that was buggy bu

I tried using them for a game server, written in C, as a test case and I came to the same conclusions. It did pretty well at making something that was buggy but worked, but increasingly continually fell apart around the 3k lines mark, which really isn't much code. It's not like these are esoteric languages, so I'm not sure why they (in my experience) have such a big problem with them. My current guess is that C and C++ aren't so mono domain. The languages its good at seems to be languages used predominantly in the web world, and most of the open code out there is for web. I'm curious if anyone has tried to use it for C# in Unity or a bespoke game in Rust and how well it does there.

starkparker 1y ago
I'm OK with an LLM not knowing how to do something. The part of working with it that I can't get past is its full-throated confidence in its actions. It'll break down, step by step, why its actions would solve the problem, then attempts to build it and can't recognize how or why its changes made the problem worse.
crq-yml 1y ago
I did spend a little while evaluating what GPT could output for a fairly intensive gaming task - implement a particular novel physical dynamics model. (Yes, there are newer tools now: I haven't used them and don't want to pay to use them.) And what it did was impressive in a certain sense in that it wrote up a large body of code with believable elements, but closer examination demonstrated that, of course, it wasn't actually going to function as-is, it made up some functions, it left others stubbed. I would never trust it to manage memory or other system resources correctly, because AFAIK nobody has implemented that kind of "simulate the program lifecycle" bookkeeping into the generated output. There's a domain that I do find some application for it in, which is "configure ffmpeg". That and tasks within game engines that are similar in nature, where the problem to be solved involves API boilerplate, do benefit. What also works, to some degree, is to ask it to generate code that follows a certain routine spec, like a state machine. Again, very boilerplate-driven stuff, and I expect an 80% solution and am pleasantly surprised if it gets everything. And it is very willing to let me "define a DSL" on the fly and then act as a stochastic compiler, generating output patterns that are largely in expectation with the input. That is something I know I could explore further. And I'm OK with this degree of utility. I think I can refine my use of it, and the tools themselves can also be refined. But there's a ceiling on it as a directly impactful thing, where the full story will play out over time as a communication tech, not a "generate code" tech, with the potential to better address Conway's law development issues.

starkparker 1y ago

I'm OK with an LLM not knowing how to do something. The part of working with it that I can't get past is its full-throated confidence in its actions. It'll break down, step by step, why its actions would solve the problem, then attempts to build it and can't recognize how or why its changes made the problem worse.

crq-yml 1y ago

I did spend a little while evaluating what GPT could output for a fairly intensive gaming task - implement a particular novel physical dynamics model. (Yes, there are newer tools now: I haven't used them and don't want to pay to use them.) And what it did was impressive in a certain sense in that it wrote up a large body of code with believable elements, but closer examination demonstrated that, of course, it wasn't actually going to function as-is, it made up some functions, it left others stubbed. I would never trust it to manage memory or other system resources correctly, because AFAIK nobody has implemented that kind of "simulate the program lifecycle" bookkeeping into the generated output. There's a domain that I do find some application for it in, which is "configure ffmpeg". That and tasks within game engines that are similar in nature, where the problem to be solved involves API boilerplate, do benefit. What also works, to some degree, is to ask it to generate code that follows a certain routine spec, like a state machine. Again, very boilerplate-driven stuff, and I expect an 80% solution and am pleasantly surprised if it gets everything. And it is very willing to let me "define a DSL" on the fly and then act as a stochastic compiler, generating output patterns that are largely in expectation with the input. That is something I know I could explore further. And I'm OK with this degree of utility. I think I can refine my use of it, and the tools themselves can also be refined. But there's a ceiling on it as a directly impactful thing, where the full story will play out over time as a communication tech, not a "generate code" tech, with the potential to better address Conway's law development issues.