5 ms·

Can you give more info on how you use/prompt those LLMs for code review and what kind of prompts you use? I've had worse experiences doing it because the quali

Can you give more info on how you use/prompt those LLMs for code review and what kind of prompts you use? I've had worse experiences doing it because the quality of answer has been quite bad, and I'm wondering if my methods are the reason.

lionkor 2mo ago
Yes, gladly! I have not yet open-sourced my skills etc., but I can give some insight and share a couple here. Review is a skill, as in, a SKILL.md with a folder full of references: - SKILL.md: https://gist.github.com/lionkor/161525be858d1d75db4c13c0f093f945 https://gist.github.com/lionkor/161525be858d1d75db4c13c0f093... - references/output-contract.md: https://gist.github.com/lionkor/8c68e33becef7a21f8408c7dc11965a6 https://gist.github.com/lionkor/8c68e33becef7a21f8408c7dc119... - references/review-lenses.md: https://gist.github.com/lionkor/0a8b080fe45306213efddf3ebb7512d9 https://gist.github.com/lionkor/0a8b080fe45306213efddf3ebb75... - references/review-workflow.md: https://gist.github.com/lionkor/d2d374b133ceb7e3660bd530ee721b2d https://gist.github.com/lionkor/d2d374b133ceb7e3660bd530ee72... - references/section-rules.md: https://gist.github.com/lionkor/8a9e503adc7fd3697410cf021f271043 https://gist.github.com/lionkor/8a9e503adc7fd3697410cf021f27... I'm aware that almost all of this is prompt voodoo, and there's no guarantee for the review to find anything or everything, but making it a dedicated skill and thoroughly observing the output thinking, tool calls, and result, lets me adjust these over time and fill the weak spots with even more prompting. I use this skill by simply telling the agent something like "Review the changes on the current branch against origin/main, take special care with backwards-incompatible changes to the public API" or something like that. I use `pi` (pi.dev) with a subagents extension, so that I can ask the agent to invoke a subagent to do the review, on work that the agent did. For models, I use the highest possible reasoning on whatever model I feel like makes sense, usually this is GPT-5.5 or deepseek flash/pro, depending on the confidentiality of the codebase, on the highest reasoning always (for reviews). I've also had success with a review checklist, though it doesn't produce an easy to parse (for humans) output: https://gist.github.com/lionkor/054ac2cf241e0765eee2383f0dba59d1 https://gist.github.com/lionkor/054ac2cf241e0765eee2383f0dba... This is why my review skill mandates a very strict output contract. I need the output to be very easy to parse, and the output contract I've specified there does that. In general I let <whatever the latest model of OpenAI's ChatGPT is> author and review SKILL.md and similar large prompts, usually with a ruleset like this, which is a 1600 line research artifact from a long GPT 5.5 "Pro" research session on prompt engineering: https://gist.github.com/lionkor/71498794d0a7d72173fc58766f250bdd https://gist.github.com/lionkor/71498794d0a7d72173fc58766f25... Does the review catch all issues? Not at all. Does it catch, usually more than one, important issue, across large changesets? Absolutely, and that's the point! :) Feel free to ask me any questions, I'm also happy to share more about my setup via email or add you or anyone else to my private repos with more of these.
- kekebo 2mo ago
  Thanks for sharing.
- rzerowan 2mo ago
  Which languages are the reviews most successful on in your work so far? Im assumining js/python , would a similar output be feasible on lower level c++ or C#/Java.
  - lionkor 2mo ago
    No js or python, it's mostly C#, Rust, and C++. They're helpful/useful on all of those, I've also used the review skill on shell scripts, GitLab pipelines, Powershell scripts, C, and a couple other odd things. It's almost never useless to run it on something, it usually finds things, even if they're sometimes not very major.

lionkor 2mo ago

Yes, gladly! I have not yet open-sourced my skills etc., but I can give some insight and share a couple here. Review is a skill, as in, a SKILL.md with a folder full of references: - SKILL.md: https://gist.github.com/lionkor/161525be858d1d75db4c13c0f093f945 https://gist.github.com/lionkor/161525be858d1d75db4c13c0f093... - references/output-contract.md: https://gist.github.com/lionkor/8c68e33becef7a21f8408c7dc11965a6 https://gist.github.com/lionkor/8c68e33becef7a21f8408c7dc119... - references/review-lenses.md: https://gist.github.com/lionkor/0a8b080fe45306213efddf3ebb7512d9 https://gist.github.com/lionkor/0a8b080fe45306213efddf3ebb75... - references/review-workflow.md: https://gist.github.com/lionkor/d2d374b133ceb7e3660bd530ee721b2d https://gist.github.com/lionkor/d2d374b133ceb7e3660bd530ee72... - references/section-rules.md: https://gist.github.com/lionkor/8a9e503adc7fd3697410cf021f271043 https://gist.github.com/lionkor/8a9e503adc7fd3697410cf021f27... I'm aware that almost all of this is prompt voodoo, and there's no guarantee for the review to find anything or everything, but making it a dedicated skill and thoroughly observing the output thinking, tool calls, and result, lets me adjust these over time and fill the weak spots with even more prompting. I use this skill by simply telling the agent something like "Review the changes on the current branch against origin/main, take special care with backwards-incompatible changes to the public API" or something like that. I use `pi` (pi.dev) with a subagents extension, so that I can ask the agent to invoke a subagent to do the review, on work that the agent did. For models, I use the highest possible reasoning on whatever model I feel like makes sense, usually this is GPT-5.5 or deepseek flash/pro, depending on the confidentiality of the codebase, on the highest reasoning always (for reviews). I've also had success with a review checklist, though it doesn't produce an easy to parse (for humans) output: https://gist.github.com/lionkor/054ac2cf241e0765eee2383f0dba59d1 https://gist.github.com/lionkor/054ac2cf241e0765eee2383f0dba... This is why my review skill mandates a very strict output contract. I need the output to be very easy to parse, and the output contract I've specified there does that. In general I let <whatever the latest model of OpenAI's ChatGPT is> author and review SKILL.md and similar large prompts, usually with a ruleset like this, which is a 1600 line research artifact from a long GPT 5.5 "Pro" research session on prompt engineering: https://gist.github.com/lionkor/71498794d0a7d72173fc58766f250bdd https://gist.github.com/lionkor/71498794d0a7d72173fc58766f25... Does the review catch all issues? Not at all. Does it catch, usually more than one, important issue, across large changesets? Absolutely, and that's the point! :) Feel free to ask me any questions, I'm also happy to share more about my setup via email or add you or anyone else to my private repos with more of these.

kekebo 2mo ago
Thanks for sharing.
rzerowan 2mo ago
Which languages are the reviews most successful on in your work so far? Im assumining js/python , would a similar output be feasible on lower level c++ or C#/Java.
- lionkor 2mo ago
  No js or python, it's mostly C#, Rust, and C++. They're helpful/useful on all of those, I've also used the review skill on shell scripts, GitLab pipelines, Powershell scripts, C, and a couple other odd things. It's almost never useless to run it on something, it usually finds things, even if they're sometimes not very major.