15 ms·
Tell HN: A new way to use GPT-3 to generate code (and everything else)
Hi HN,
One of the things that frustrates me about Copilot is that all tasks posed to it must be in the form of a completion. By writing clever comments you can get it to generate a few lines of code or a short function body, but you never get coherent long-form generations just from mashing the tab key.
I’m working on a different approach. Instead of requiring you specify your code generation task through stilted comments, you can use GPT-3 to fill in what I call “instructional templates”. They’re like f-strings, except the English goes on the inside and the Python goes on the outside. Additionally, each instruction’s location and surrounding context can aid in interpreting it, allowing instructions to be impressively terse.
I’ve collected 10 examples of the method on a Twitter thread here. Most code examples are in Python, but I also demonstrate generating CSV, NDJSON, R, Markdown, and HTML: https://twitter.com/goodside/status/1559801520773898240?s=21&t=-r-dR8pkhZ3lfCpeLOWqvw https://twitter.com/goodside/status/1559801520773898240?s=21...
I also have a few examples of more creative, non-program output in HTML and Markdown in this thread: https://twitter.com/goodside/status/1560953991722418177?s=21&t=-r-dR8pkhZ3lfCpeLOWqvw https://twitter.com/goodside/status/1560953991722418177?s=21...
Interested in any feedback, especially from anyone who’s tried to apply my method to their own problems.
- potatoman22 4y agoYou should make this into a website generation playground
- bredren 4y agoReminds me some of Django or jinja template tags. Except the tag names are free form and the source is generated rather than defined.
- deleted 4y ago[deleted]
- goodside 4y agoClickable version of links: Python, CSV, NDJSON, R, Markdown, and HTML examples: https://twitter.com/goodside/status/1559801520773898240?s=21&t=-r-dR8pkhZ3lfCpeLOWqvw https://twitter.com/goodside/status/1559801520773898240?s=21... More creative, non-program output in HTML and Markdown: https://twitter.com/goodside/status/1560953991722418177?s=21&t=-r-dR8pkhZ3lfCpeLOWqvw https://twitter.com/goodside/status/1560953991722418177?s=21...
- zaroth 4y agoIsn’t there a different domain I can swap in so I can actually read this in a browser without immediately getting the Twitter “paywall” / login prompt?
- sltkr 4y agoTry replacing twitter.com with nitter.net or twiiit.com in the URL. The latter is a proxy for various Nitter instances, which is useful if nitter.net itself is overloaded, which is happening more and more these days because obviously it's such a useful service.
- jonnycomputer 4y agonitter.net
- deleted 4y ago[deleted]
- easrng 4y agoReplace https://twitter.com/ https://twitter.com/ with https://farside.link/nitter/ https://farside.link/nitter/ to get redirected to a working nitter instance.
- lifthrasiir 4y agoIn this case everything is in images, so the following should work: https://pbs.twimg.com/media/FaWHN2oXkAAPISD.jpg https://pbs.twimg.com/media/FaWHN2oXkAAPISD.jpg https://pbs.twimg.com/media/FaWHN2nWQAEERHQ.jpg https://pbs.twimg.com/media/FaWHN2nWQAEERHQ.jpg https://pbs.twimg.com/media/FaWHN2pWYAIbJJ4.jpg https://pbs.twimg.com/media/FaWHN2pWYAIbJJ4.jpg https://pbs.twimg.com/media/FaWHN2oX0AACDC6.jpg https://pbs.twimg.com/media/FaWHN2oX0AACDC6.jpg https://pbs.twimg.com/media/FaYtV6WXEAkTG83.jpg https://pbs.twimg.com/media/FaYtV6WXEAkTG83.jpg https://pbs.twimg.com/media/FaYtV6SX0AMHbYh.jpg https://pbs.twimg.com/media/FaYtV6SX0AMHbYh.jpg https://pbs.twimg.com/media/FaYtV6YXkAE0FtH.jpg https://pbs.twimg.com/media/FaYtV6YXkAE0FtH.jpg https://pbs.twimg.com/media/Fac0K14aAAIdU5p.jpg https://pbs.twimg.com/media/Fac0K14aAAIdU5p.jpg https://pbs.twimg.com/media/Fac0K13UEAEb2eq.jpg https://pbs.twimg.com/media/Fac0K13UEAEb2eq.jpg https://pbs.twimg.com/media/Fac0K1xUEAY_ZAR.jpg https://pbs.twimg.com/media/Fac0K1xUEAY_ZAR.jpg https://pbs.twimg.com/media/Fac0K19akAUgzk1.jpg https://pbs.twimg.com/media/Fac0K19akAUgzk1.jpg https://pbs.twimg.com/media/Fai8ou4WAAApNaD.jpg https://pbs.twimg.com/media/Fai8ou4WAAApNaD.jpg https://pbs.twimg.com/media/Fai8ou3WIAU2AOs.jpg https://pbs.twimg.com/media/Fai8ou3WIAU2AOs.jpg https://pbs.twimg.com/media/Fai8ovXVUAAE2K_.jpg https://pbs.twimg.com/media/Fai8ovXVUAAE2K_.jpg https://pbs.twimg.com/media/FajBEnPXwAAceIK.jpg https://pbs.twimg.com/media/FajBEnPXwAAceIK.jpg https://pbs.twimg.com/media/FajBEnuVsAA4885.jpg https://pbs.twimg.com/media/FajBEnuVsAA4885.jpg https://pbs.twimg.com/media/FajBEnOWQAA9x7C.jpg https://pbs.twimg.com/media/FajBEnOWQAA9x7C.jpg https://pbs.twimg.com/media/FajBEnwUcAADGMX.jpg https://pbs.twimg.com/media/FajBEnwUcAADGMX.jpg https://pbs.twimg.com/media/FajjhfWX0AAVBNg.jpg https://pbs.twimg.com/media/FajjhfWX0AAVBNg.jpg https://pbs.twimg.com/media/FajjhfTWQAED_kB.jpg https://pbs.twimg.com/media/FajjhfTWQAED_kB.jpg https://pbs.twimg.com/media/FajjhfUWAAApBjT.jpg https://pbs.twimg.com/media/FajjhfUWAAApBjT.jpg https://pbs.twimg.com/media/FajjhfVXoAAKoK-.jpg https://pbs.twimg.com/media/FajjhfVXoAAKoK-.jpg https://pbs.twimg.com/media/FalEEs-WAAAHo_J.jpg https://pbs.twimg.com/media/FalEEs-WAAAHo_J.jpg https://pbs.twimg.com/media/FalEEs-WIAEtrUW.jpg https://pbs.twimg.com/media/FalEEs-WIAEtrUW.jpg https://pbs.twimg.com/media/FalEEs_WIAAIAlP.jpg https://pbs.twimg.com/media/FalEEs_WIAAIAlP.jpg https://pbs.twimg.com/media/FalQzNPX0AAFUvr.jpg https://pbs.twimg.com/media/FalQzNPX0AAFUvr.jpg https://pbs.twimg.com/media/FalQzNNWYAAj4ed.jpg https://pbs.twimg.com/media/FalQzNNWYAAj4ed.jpg https://pbs.twimg.com/media/FalQzNOXkAEFB5F.jpg https://pbs.twimg.com/media/FalQzNOXkAEFB5F.jpg https://pbs.twimg.com/media/Falk4AyXEAArsb7.jpg https://pbs.twimg.com/media/Falk4AyXEAArsb7.jpg https://pbs.twimg.com/media/Falk4AyXgAEQUeO.jpg https://pbs.twimg.com/media/Falk4AyXgAEQUeO.jpg https://pbs.twimg.com/media/Fal0dGKX0AIK4w6.jpg https://pbs.twimg.com/media/Fal0dGKX0AIK4w6.jpg https://pbs.twimg.com/media/Fal0dGGWQAABXxd.jpg https://pbs.twimg.com/media/Fal0dGGWQAABXxd.jpg https://pbs.twimg.com/media/Fal0dGKXoAAiTqC.jpg https://pbs.twimg.com/media/Fal0dGKXoAAiTqC.jpg https://pbs.twimg.com/media/Fal0dGIWQAAo3GU.jpg https://pbs.twimg.com/media/Fal0dGIWQAAo3GU.jpg
- fotcorn 4y agoSeems like this also works with C++. This was my first time using GPT3, and first attempt at this, no fine-tuning, no cherry picking at all: Replace the $$dollar-quoted expressions$$ with the content described therein, or follow the directions they contain. main.cpp ``` $$c++ includes for output$$ $$Vec3 struct with three float members x, y, z. constructor. dot product implementation. $$ $$main method which allocates two vectors, then prints the dot product of them to standard output$$ ``` Answer: main.cpp ``` #include <iostream> struct Vec3 { float x, y, z; Vec3(float x, float y, float z) : x(x), y(y), z(z) {} float dot(const Vec3& v) const { return x * v.x + y * v.y + z * v.z; } }; int main() { Vec3 a(1, 2, 3); Vec3 b(4, 5, 6); std::cout << a.dot(b) << std::endl; } ```
- goodside 4y agoNice! In general these are better if you run them at the lowest possible temperature. I.e., try temp=0 first for deterministic output and then raise slowly if you need to cherry-pick a better generation.
- wyldfire 4y ago> Replace the $$dollar-quoted expressions$$ with the content described therein, or follow the directions they contain. Is this a natural language meta-instruction prompt to GPT-3? If so, that seems kinda impressive. Does the model conceptualize this sentence somehow, or 'merely' recognize similar prompts from some specialized training?
- sp332 4y agoHere's one of the craziest examples I've seen. It's just a page of instructions, and GPT-3 follows them https://mobile.twitter.com/goodside/status/1557524546412052482 https://mobile.twitter.com/goodside/status/15575245464120524...
- deleted 4y ago[deleted]
- yomkippur 4y agoshould i cancel my sub to copilot? alright how do i access gpt3 so i can write it like this?
- goodside 4y agoThe OpenAI API. I’m using text-davinci-002 (the default) for all of these, with temperature=0 for reproducibility/quality.
- CamperBob2 4y agoMore like, should I withdraw my college application, or at least switch away from CS. Obviously the answer is not only "No", but "Hell, no, this is just starting to get interesting!" But a lot of young people who are primarily in it for the coding career prospects should probably reconsider.
- ImHereToVote 4y agoSo only people who have rich parents should go into CS. Got it
- CamperBob2 4y agoFor grossly-invalid values of "Got it."
- sltkr 4y agoPosting screenshots to Twitter has to be the least convenient way to share code online (short of actual trolling).
- goodside 4y agoI include OpenAI Playground links for all but the first several of these, which capture not only the exact prompt but the settings used in the generation. I don’t use gists because you need multiple, non-contiguous highlighted spans of text. Also the Playground formatting is recognizable in a way that establishes context quickly and invites people to read the text.
- fuzzythinker 4y agoFYI, I never click twitter links, that's 1 data point. Not sure how many are like me here.
- goodside 4y agoThanks. I appreciate not everyone likes or is willing to use Twitter, but I’ve yet to find a more convenient or accessible channel for this content. I could start a proper blog and worry about my own prompt formatting, but that complicates my workflow a lot — especially since most of these results come directly from my phone in my spare/travel time. Eventually I may consolidate my better findings into a blog post of some kind.
- thenthenthen 4y agoUse open ai to generate the blog for you! :) anyway thanks for posting it here and not only on Twitter. Curious to learn more now
- dang 4y ago"Please don't complain about tangential annoyances—things like article or website formats, name collisions, or back-button breakage. They're too common to be interesting." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- terafo 4y agoIs it in your plans to do such thing with actually open model?(something like Codegen from Salesforce or BLOOM).
- goodside 4y agoNo. The method relies heavily on the peculiar fine-tuning of the InstructGPT line of models, which are trained specifically to follow MTurk-style prose instructions. I imagine achieving similar results using a non-InstructGPT model would be hard, but I could be wrong.
- renewiltord 4y agoThis is a very clever piece of prompting. Thank you for the idea. Great discovery!
- brunooliv 4y agoIs co-pilot now mandatory to be paid? I used a trial for a long while, at least enough to miss it in my workflows once it went paid. Is this really the case? Last I checked it was like 12 dollars a month? Is it possible for it still to be free or are there free plug-ins for intellij that leverage it? I loved it for some scaffolding, quick drafts or explore some language features, but, not enough to pay monthly for it (yet!)
- coding123 4y agoThe most comical thing about it going paid was all the hit pieces about it went away.
- kortilla 4y agoNo, they are still there. Nothing has changed. Why would there be new ones if nothing has changed? GitHub (msft) is still just copy pasting code from other projects with no regard for copyright. Whether it’s paid or not doesn’t matter much.
- brunooliv 4y agoIt matters to me for my personal use cases and basically for its scaffolding capabilities, because, while I was finding it useful and interesting, I don't think it's worthy of a monthly sub. What the comment above me meant is that probably this is a shared feeling by many other people because the initial hype (like all the blog posts etc) died quite fast once it went paid. I can't imagine how much it cost to train the thing but oh well, I guess it's irrelevant for me for now since I won't be using it...
- avinassh 4y agoWow, I had no idea this could be done! Since we are on the topic of code generation, I had a question. I built this joke script called Rockstar [0] which generates fake git commits resulting in fully green GitHub commit graph. In each commit it adds gibberish, in the last commit adds a valid code. I wanted to know if there’s an easy way to generate realistic looking code which I can use in each commit? I can’t expect users of the script to use OpenAI or any such API service. Something which can be used to generate code locally would be sweet! [0] - https://github.com/avinassh/rockstar https://github.com/avinassh/rockstar
- HasanYousef 4y agoThis feels, not ethical.
- O__________O 4y agoMight be worth explicitly staying you feel it’s unethical so it’s possible to respond to your position. I for one am able to think of numerous ethical ways it might be of use.
- trailingComma 4y agoCould you share them?
- andy_xor_andrew 4y agoethical use: you use it, and the squares on your github.com profile, which is a website that hosts git repos, turn green.
- O__________O 4y agoWhile I would be the first to say those squares are meaningless without context, if someone led others to believe they were the result of anything other than automatically generated posted and didn’t have a legitimate reason like the one I posted, that might potentially be fraud, especially if it was used as a basis for future economic exchanges, such as employment.
- snapcaster 4y agoThis is really cool, commenting so i can easily find this later to check out
- O__________O 4y ago(Unless your comment adds to the thread, please get familiar with the how your profile links to “upvoted submissions” or “favorite submissions” work — or find another way to log post of interest to you; other than a generic comment for yourself.)
- howon92 4y agoThanks for sharing this. I've been playing around with GPT-3 for a bit. Have you tried comparing this method to using the `insert` mode in the Playground? https://beta.openai.com/playground?mode=insert https://beta.openai.com/playground?mode=insert On a side note, I learned that the limitation on the number of tokens was often too restrictive to do anything fun with code generation. Have you run into this issue too?
- goodside 4y agoThe Insert API is much less powerful, because you can infill only a single location and you’re limited to communicating the infill content purely through context, without any instruction. The Edit API is more directly adaptable to this, and anecdotally it does seem to work but I can’t vouch for its reliability.
- andreyk 4y agoThis is using standard prompting I think? It'd be neat to try with the full 'fill in the blank' (where the blank is in the middle of the input) generation technique LLMs can support, might work even better!
- goodside 4y agoThis definitely isn’t standard prompting — as far as I know, nobody was aware of this technique until I found it this past week. Its precedent is the “format trick” that Boris Power (at OpenAI) showed me, which is essentially my “instruction templates” but where the only template form is an uninformative numbered list. The trick is used purely as a way to consistently format the output to multi-part tasks without providing fully worked examples. As far as I know, it’s never been extended to contextually informative templates, or to achieve code/document synthesis, as I’ve done here. An example of the “format trick”, roughly as Boris described it to me, though in OpenAI’s examples the answers were simply numbered: https://twitter.com/goodside/status/1556729629263933440?s=21&t=s4JVKNOb4xONEQqT3nXlXQ https://twitter.com/goodside/status/1556729629263933440?s=21... If someone at OpenAI corrects me and says they’ve known about my method, I’ll stop claiming to have discovered it.
- ricardobayes 4y agoJust as cybersecurity analyst jobs are getting reduced to comparing risk score numbers, maybe programming jobs will be 'reviewing' machine-generated code in the future.
- janekm 4y agoI think it will be more like pair programming. Already kind of feels like you’re pair programming with an intern with copilot.
- simonw 4y agoI've been following Riley on Twitter and he's a constant source of fantastic GPT-3 tips, recommended: https://twitter.com/goodside https://twitter.com/goodside
- goodside 4y agoThanks! Your blog post on using GPT-3 dialog to explain SQL queries was a big inspiration for me to start posting my prompts publicly: https://simonwillison.net/2022/Jul/9/gpt-3-explain-code/ https://simonwillison.net/2022/Jul/9/gpt-3-explain-code/
- overview 4y agoI just copy and pasted 3 random Leetcode problem prompts to GPT-3. It successfully generated Python code that passed all test cases for 2 out of the 3. Problems passed: - Two sum - Text Justification edit: Newlines
- b20000 4y agoif you take a coding interview and answer the question by feeding it into GPT-3, does that mean you pass the interview? it must do really well since all it takes is memorizing the solutions to a large body of meaningless challenges. the implication here is that if GPT-3 can solve your coding question, you are hiring people good at memorizing solutions and not skillfull engineers.
- Melting_Harps 4y ago> the implication here is that if GPT-3 can solve your coding question, you are hiring people good at memorizing solutions and not skillfull engineers. How is that any different than what it is now? Moreover, this type of rote learning is what has been rewarded most in most academic examination processes; SAT prep tests are doing exactly that: cramming students in order learn the exam process and deductive reasoning via multiple choice by taking many test exams. Honestly, the only tech job I have ever received was extended to me because I was headhunted while working on my startup so my POV vastly differs from most with how on-boarding should go, and every time I see this white-boarding non-sense is the norm in the interview process it make me wonder what exactly it's trying to achieve if not just gate-keep and subject people to the same hazing process the hiring managers and project leads went through when they joined: people just say they'll just grind on leetcode rather than build something useful as this is the main barrier of entry and it dissapoints me so much that it's just been normalized over the years.
- lupire 4y agoClearly false. Humans do not solve problems the same way computers do.
- infogulch 4y agoWhat I would really like is to get it to craft commit diffs of an existing codebase in response to a request.
- deleted 4y ago[deleted]
- maytc 4y agoGPT-3 also understand code to some degree so it can run the code as instructed. https://mayt.substack.com/p/gpt-3-can-run-code https://mayt.substack.com/p/gpt-3-can-run-code
- benreesman 4y agoI find it fascinating that it seems like there's this emerging field of expertise around how to best interact with these gigantic LMs. I have no idea if this is like a passing fad during a gawky adolescent phase of the models, or if this is just a new thing that some people will always be at the cutting edge of. I gather that there's loose some precedent for the latter with the chess computer stuff: I've read that serious chess players heavily incorporate computers into their training and even that hybrid human/computer teams often outperform either/or teams. I'd love if someone who actually knows chess commented. Generating code via model sampling seems to have different, or at least exaggerated imperatives around "few-shot" tuning. One might wish to generate natural language for any number of purposes, but there is a probably a stronger "better"/"worse" gradient for code, and much like human language, excellent code is rare as a fraction of all code not only overall, but even by the same company or even by the same author. So you probably want the overall contours of like "this code compiles" from a big corpus, but to tune up the "this vectorizes well" eigen-tensor from Lemire's repo. Crazy times.
- int_19h 4y agoIt reminds me of Vernor Vinge's "Rainbows End", where one of the basic classes they have in school is how to correctly word your search queries for best results. We might actually end up with a fancier version of that.
- NhanH 4y agoHuman absolutely bring negative value to a chess match for at least 10 years now (when pairing with a computer). Even for Go AI, where it is only ~6 years old now, I'd still believe that human bring 0 value to a computer. 2-3 years ago, there was still some precise sequence where human might conceivable be dealing with (due to the nature of MCTS, human was better at reading deep narrow move sequences -- in go player term, human was sometimes better at life & death + some ladder sequences). Nowadays, the network are just way stronger than any human that we can't really follow what is going on in serious hour-long matches of the machine.
- qzw 4y ago> Human absolutely bring negative value… I have people skills; I am good at dealing with people. Can't you understand that? What the hell is wrong with you people?