3 ms·

Yea I'm totally with you! In fact, when I started Second, I was resisting GPT-3, and had planned on using a pure deterministic compiler. However, through a lo

Yea I'm totally with you! In fact, when I started Second, I was resisting GPT-3, and had planned on using a pure deterministic compiler. However, through a lot of experimentation, I discovered that the OpenAI code edit APIs, when using a temperature value of 0 (deterministic) provides shockingly good and fairly consistently results. I found that I get the best results when combining my deterministic compilers with GPT-3 The cool thing about Second is that the handoff between bot and human is GitHub. In the tool today, changes are committed directly to the repo, however very soon the default behavior will be pull requests. This is great because it gives engineers the opportunity to see the diffs and make tweaks before merging in.

fuzzythinker 4y ago
About how soon is "very soon"?
- immortalloom 4y ago
  Lol well, I planning on making it the default last night! But I ran into some serious production issues (GitHub API calls suddenly started failing) so I got side tracked. If enough folks ask for it, I could prioritize it and make the code change in like an hour (mostly testing)
bluecoconut 4y ago
Small warning: 0 temperature still isn't deterministic with openAI endpoints. So, if you are relying on this as an absolute, it'll definitely fail at some point (this bit me at one point recently). If you just like it because it's "mostly deterministic" so it feels more well behaved, then it's fine. From here: https://platform.openai.com/docs/guides/completion https://platform.openai.com/docs/guides/completion > Setting temperature to 0 will make the outputs mostly deterministic, but a small amount of variability may remain.
- immortalloom 4y ago
  Yea definitely, I've noticed that a zero temperature isn't perfectly deterministic, but the results seem reasonably consistent. I will honestly say that there have been a couple of wtf file generations, where even with a temperature of 0, GPT-3 created something I did not expect. I'd like to enable engineers to run a retry on specific files right from GitHub, like making comments in the PR itself.

fuzzythinker 4y ago

About how soon is "very soon"?

immortalloom 4y ago
Lol well, I planning on making it the default last night! But I ran into some serious production issues (GitHub API calls suddenly started failing) so I got side tracked. If enough folks ask for it, I could prioritize it and make the code change in like an hour (mostly testing)

bluecoconut 4y ago

Small warning: 0 temperature still isn't deterministic with openAI endpoints. So, if you are relying on this as an absolute, it'll definitely fail at some point (this bit me at one point recently). If you just like it because it's "mostly deterministic" so it feels more well behaved, then it's fine. From here: https://platform.openai.com/docs/guides/completion https://platform.openai.com/docs/guides/completion > Setting temperature to 0 will make the outputs mostly deterministic, but a small amount of variability may remain.

immortalloom 4y ago
Yea definitely, I've noticed that a zero temperature isn't perfectly deterministic, but the results seem reasonably consistent. I will honestly say that there have been a couple of wtf file generations, where even with a temperature of 0, GPT-3 created something I did not expect. I'd like to enable engineers to run a retry on specific files right from GitHub, like making comments in the PR itself.