3 ms·

I’m not sure I agree. Over the last few days I’ve been cobbling together a little service which asks GPT to identify issues with code, then if it can generate

I’m not sure I agree. Over the last few days I’ve been cobbling together a little service which asks GPT to identify issues with code, then if it can generate solutions, the service attempts to apply the solution and run the code. I’ve been learning to dial in prompts and finding this truly is a major factor of the job that needs to be just right. I went from unreliable response formats, hallucinated solutions, naive solutions, and so on all the way to fairly conservative, pragmatic, consistent responses. My success rate isn’t as high as I’d like, but I find it absolutely remarkable. My key takeaway so far has been that a lot of perceived limitations are in fact a lack of exposure to what effective prompting makes possible. Another feature that’s powerful yet overlooked is what recursive prompting can accomplish. This is what makes me think complexity might be overcome in the not-so-distant future. For example, if I apply a solution and it breaks something else, I can then reform a prompt based on the previous prompt along with a new one informing it of what went wrong. These little recursive exchanges have actually yielded useful results in multiple tests. That’s a huge deal in my opinion; this is totally automated by a smooth brain like me. As token limits increase and the models become more capable, I suspect this tool will actually improve without me changing much. I suspect it could still work quite a bit better and more efficiently if I could more succinctly provide context about the code in my prompts, too. That would overcome token limitations for the time being and likely improve accuracy quite a bit. So, if things like this are possible already… I have a feeling we’re going to overcome the complexity issue sooner than it currently seems.

turtleyacht 4y ago
> dial in prompts > effective prompting... recursive prompting It feels like you are massaging a giant library of macros :) The way you describe interacting with GPT by iterative attempts, constructing working primitives is pretty much programming, right? AI might get more folks into computer science, or even programming; more hackers, more inquisitive minds, and maybe--just maybe--this "assault on complexity" earns some real victories.
adave 4y ago
So you are converting your code into english and then asking GPT to convert it back to code? Might as well right the code at that point.
- steve_adams_86 4y ago
  No, I send the code causing the error as part of the prompt, and it responds with that code with corrections (if it can). The only English is the prompt describing the role and response I need from GPT. I could write the code myself, but the thing is, if this program works then in theory it could address trivial errors at any time of day, at very high speed. The better it gets at doing this, the more useful it could be. Most automations start out very inefficient, and many stay that way. I mostly do it to learn but I also see a real opportunity to remove a certain type of task from my plate if this service works well. Of course, other companies (like GitHub) have likely already prototyped this and whoever runs with it will do a far better job than I'm doing. That's totally fine though, I'm just doing this to learn. At first I was thinking I would have this inspect errors from something like Sentry, try to solve the issue, then spin up a PR or something if it succeeded (I have a version which also writes tests to verify its changes succeeded, but it's flaky when the intent in the code isn't crystal clear). I realized though that this would probably be better as an extension in an IDE which can intercept error stacks in a terminal or over a port. That way there's full control over apply patches and no need to arbitrarily run code, which requires a whole virtual machine approach. Anyway, fun to learn, might be useful somehow, but totally cool if ultimately I do just write the code myself.

turtleyacht 4y ago

> dial in prompts > effective prompting... recursive prompting It feels like you are massaging a giant library of macros :) The way you describe interacting with GPT by iterative attempts, constructing working primitives is pretty much programming, right? AI might get more folks into computer science, or even programming; more hackers, more inquisitive minds, and maybe--just maybe--this "assault on complexity" earns some real victories.

adave 4y ago

So you are converting your code into english and then asking GPT to convert it back to code? Might as well right the code at that point.

steve_adams_86 4y ago
No, I send the code causing the error as part of the prompt, and it responds with that code with corrections (if it can). The only English is the prompt describing the role and response I need from GPT. I could write the code myself, but the thing is, if this program works then in theory it could address trivial errors at any time of day, at very high speed. The better it gets at doing this, the more useful it could be. Most automations start out very inefficient, and many stay that way. I mostly do it to learn but I also see a real opportunity to remove a certain type of task from my plate if this service works well. Of course, other companies (like GitHub) have likely already prototyped this and whoever runs with it will do a far better job than I'm doing. That's totally fine though, I'm just doing this to learn. At first I was thinking I would have this inspect errors from something like Sentry, try to solve the issue, then spin up a PR or something if it succeeded (I have a version which also writes tests to verify its changes succeeded, but it's flaky when the intent in the code isn't crystal clear). I realized though that this would probably be better as an extension in an IDE which can intercept error stacks in a terminal or over a port. That way there's full control over apply patches and no need to arbitrarily run code, which requires a whole virtual machine approach. Anyway, fun to learn, might be useful somehow, but totally cool if ultimately I do just write the code myself.