4 ms·
I've experimented a bit with this on the raw Codex model (https://smitop.com/post/codex/ https://smitop.com/post/codex/), and I've found that some prompt engine
by smitop 5y ago
I've experimented a bit with this on the raw Codex model (https://smitop.com/post/codex/ https://smitop.com/post/codex/), and I've found that some prompt engineering can be helpful: explicitly telling the model to generate secure code in the prompt sometimes helps. (such as by adding to the prompt something like "Here's a PHP script I wrote that follows security best practices"). Codex knows how to write more secure code, but without the right prompting it tends to write insecure code (because it was trained on a lot of bad code).
> the settings and documentation as provided do not allow users to see what these are set to by default
There isn't a single default value. Those parameters are chosen dynamically (on the client side): when doing more sampling with a higher top_p a higher temperature is used. I haven't tracked down where the top_p value is decided upon, but I think it depends on the context: I believe explicitly requesting an completion causes a higher top_p and a more capable model (earhart), which gives better but slower results than the completions you get as autocomplete (which are from the cushman model with a lower top_p). Copilot doesn't use any server-side magic, all the Copilot servers do is replace the GitHub authentication token with an OpenAI API key and forward the request to the OpenAI API.
- kiwih 5y ago> I've found that some prompt engineering can be helpful: explicitly telling the model to generate secure code in the prompt sometimes helps. As noted in the diversity of prompt section, we did try a lot of different/ reasonable changes to the prompt to see what would happen in our SQL injection scenario. In our case, asking it to make it secure actually made the prompt slightly worse (!), and the biggest bias towards making the code better was having other good code. > There isn't a single default value. That's what we also guess, but as you say, it's not written or documented anywhere.
- SrslyJosh 5y ago> Codex knows how to write more secure code I think that "knows" is anthropomorphizing too much--you're giving hints that cause the model to use a more secure subset of its training data. =)
- wly_cdgr 5y agoI think this is algomorphizing what humans do when given such a hint too much :)