45 ms·
Using GPT-3 to explain how code works
- jwilk 4y ago> The (?xm) notation is a modifier that allows the regular expression to span multiple lines and to use comments. This is incorrect. (?xm) enables the VERBOSE (X) and MULTILINE (M) flags. From the docstring: M MULTILINE "^" matches the beginning of lines (after a newline) as well as the string. "$" matches the end of lines (before a newline) as well as the end of the string. X VERBOSE Ignore whitespace and comments for nicer looking RE's.
- xdfgh1112 4y agoThe X flag also allows the written regex to span multiple lines, i.e. it counts newlines as whitespace. Interesting confusion there.
- simonw 4y agoHah yeah, good catch - I'd missed that incorrect "allows it to span multiple lines" statement.
- bo1024 4y agoThe scary part about good AI is not knowing how good it is, i.e. not being able to trust it because you don't know when it'll be wrong (nor does the AI itself know).
- solardev 4y agoSame could be true of human programmers though. Except those on HN, of course ;)
- pornel 4y agoThis as an assistant/teacher could be incredibly empowering for people learning to program. I'm sceptical of AI writing programs for people, but I can easily imagine such AI guiding/unblocking non-programmers enough for them to actually program.
- layer8 4y agoIt’s a double-edged sword. From the article: GPT-3 doesn’t actually know anything about anything at all. It’s a huge pattern generator. You can’t trust anything it says, because all it does is group words together into convincing looking shapes based on text that it’s seen before. Once again, I’m reminded that tools like GPT-3 should be classified in the “bicycles for the mind” category. You still have to know how to pedal! They’re fantastic tools for thinking, but to actually use their output effectively requires VERY deep knowledge—both of the subject matter in question, and of the way that the AI tools themselves work.
- l33t2328 4y agoWhat is thinking if not pattern generating?
- trention 4y agoIt's not enough to state that the human mind is a pattern matcher as pretty much everyone of the "scale is all you need" crowd does. You have to prove it too. Otherwise it's just a variant of Russell's teapot.
- abrax3141 4y agoEven if you prove it, it's not useful.
- abrax3141 4y agoYou aren't actually serious, are you?
- Swenrekcah 4y agoWe use far more variety of inputs than GPT-3. GPT-3 has more textual training data than any human could read in a thousand lifetimes, but we base our judgment on much more than that.
- layer8 4y ago
- booleandilemma 4y agoCould they do this for legal documents too? I'd hate to see just the programmers put out of work by GPT-3.
- Transfinity 4y agoA buddy of mine works at a startup doing exactly this: https://www.advocat.ai/ https://www.advocat.ai/ Unsure if they use GPT-3 specifically, but the core idea is the same.
- punnerud 4y agoThis would be great for blind people, or you could program while biking.
- ctoth 4y agoAs a blind programmer I'm trying to understand how I should find this more useful than anyone else?
- netsec_burn 4y agoI think what they're saying is, you can quickly understand the contents of a file by having it summarized for you audibly. Versus having each token read out to you. Then you can locate the part that you're trying to program faster.
- hertzrat 4y agoNot sure what that has to do with being blind except the audio part, and we already have text to speech systems to read the actual code Programming while biking with ai is like relying on a really buggy early stage auto-drive ai system to take you places down winding, slippery roads while you sleep at the wheel. It might get you there most of the time, but I’d might bend you around a tree too
- netsec_burn 4y agoIf you're a blind programmer, would you rather listen to a summary of each function in a source file or listen to every token on 10x speed and figure it out yourself? I believe that is the point that they are making. Not sure about the programming while biking though, I typically need to concentrate on one task at a time (which is what most multitasking is realistically, just quickly switching between one task at a time).
- hertzrat 4y agoAgain, what does this have to do with being blind? Whether you can see or not, summaries are nice to have; and whether you can see or not, you still need to read the code in case the summary isn’t accurate, as it typically isn’t
- f38zf5vdt 4y ago> As with everything GPT-3, this answer sounds very convincing! But is this actually true? I would expect the SQL query optimizer to be smart enough to optimize the LEFT JOIN to the point where it would run efficiently. I think GPT-3’s answer to that question is actually very misleading. I really enjoy how well it gives wrong answers. The internet being polluted with GPT3-like text models is going to send us back to the time when verifying anything online as truthful is difficult, if it hasn't already.
- mwint 4y agoMaybe my next move should be to start a consulting shop: “We Fix GPT-3 Code”.
- abrax3141 4y agoGood luck with that. I don't think that these things are fixable. They're basically giant grammatical confabulation engines. When you want grammatically-correct random noise, great. Don't fly a plane with them, please...or at least not one I'm on!
- trention 4y agoHe meant fixing the output(s) not the model that produced them.
- atq2119 4y agoI'm pretty sure abrax3141 understood this. Some things are just unfixable because they're "not even wrong". You basically have to start from scratch.
- goodside 4y agoNo method is fool-proof for GPT-3, but there are tricks that suppress its tendency to hallucinate, such as demonstrating in a k-shot prompt the possibility of false assumptions in a question. I demonstrate this here: https://mobile.twitter.com/goodside/status/1545793388871651330 https://mobile.twitter.com/goodside/status/15457933888716513... Models more advanced that GPT-3, such as LaMDA, have entire subsystems specifically dedicated to “grounding” output in truthful information. Hallucination is at least partially a solved problem, but the methods haven’t disseminated broadly yet.
- layer8 4y agoTangential question: Is GPT-3 (and similar tools) being kept up to date with new information? Or would it basically have to be retrained from scratch to properly incorporate new data?
- simonw 4y agoThere's a mechanism you can use to add your own training data on top of GPT-3. This tutorial talks about using that to generate song lyrics (about half way through): https://wingedsheep.com/songwriting-with-gpt-3/ https://wingedsheep.com/songwriting-with-gpt-3/ I don't know how often OpenAI build new versions of the core GPT-3 language model.
- abirch 4y agoWhat would be highly beneficial is for the AI to understand changes so when it spots an old pattern in my code, it could suggest the new pattern. I'd pay for a Copilot that did that.
- oefrha 4y agoCopilot: “You’re using yesterday’s JavaScript framework of the day. Please rewrite your app in today’s JavaScript framework of the day.” Jokes aside, I wonder when we’ll see an official language linter (go vet, rust-clippy, etc.) that is an ML black box instead of a traditional handcrafted deterministic rule-based engine.
- loufe 4y agoThis is fantastic, I've been enjoying myself thoroughly with Dall-E 2 recently and similar to this love trying to imagine how the training method / training data must have led to the output. It's truly amazing times we live in, but it will take a lot more work than people 70 years ago thought to get to a level of polish.
- akamoonknight 4y agoThis is a fascinating use case that wouldn't have occurred to me at all, quite cool, even if it "doesn't know anything about anything at all". It even does pretty good with Verilog, which I assumed it would have issues with, admittedly some of this is just "google-able", but even so. It can even create a reasonable attempt at a testbench (even though it doesn't quite get it right), which is crazy to me. *Explain this code: module my_module(a,c,y); input a,c; output y; assign y= c ? a : 1'bz; endmodule* This code is a Verilog module that defines three inputs (a, c, y) and one output (y). The output y is assigned to the value of a if c is true, otherwise it is assigned to 1'bz. *What is 1'bz ?* 1'bz is a Verilog constant that represents a bit value of 'z', which is used to indicate a high impedance state. *Why is high impedance used ?* High impedance is used in digital circuits to allow multiple drivers to share a single bus without contention. When two or more drivers attempt to drive the same bus at the same time, a contention can occur and the bus voltage can become undefined. By using high impedance, drivers can avoid contention and allow the bus to remain stable. *Give me a testbench for this module* module testbench; reg a; reg c; wire y; my_module dut(a, c, y); initial begin a = 1'b0; c = 1'b0; #1; a = 1'b1; c = 1'b1; #1; a = 1'bz; c = 1'b1; #1; a = 1'bx; c = 1'b1; #1; $stop; end endmodule *Create the same testbench but add validation of the output.* module testbench; reg a; reg c; wire y; my_module dut(a, c, y); initial begin a = 1'b0; c = 1'b0; #1; a = 1'b1; c = 1'b1; #1; a = 1'bz; c = 1'b1; #1; a = 1'bx; c = 1'b1; #1; $stop; end always @(*) begin if (c == 1'b0) assert(y == 1'b0); else if (c == 1'b1) assert(y == a); else if (c == 1'bz) assert(y == 1'bz); else if (c == 1'bx) assert(y == 1'bx); end endmodule
- hackernewds 4y agoThis is frighteningly impressive
- electrondood 4y agoThis is already starting to feel like the Star Trek voice computer interface. You ask it questions, follow-up questions about anything.
- deleted 4y ago[deleted]
- goodside 4y agoFor the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output. Many of these are discussed in the Twitter thread the OP links to, but I'd like to highlight a few of my favorites here too. - GPT-3 can produce simple SVG diagrams (https://twitter.com/goodside/status/1545883023719239689 https://twitter.com/goodside/status/1545883023719239689) from an English description, or incrementally modify a given SVG diagram (https://twitter.com/goodside/status/1545888869207400454 https://twitter.com/goodside/status/1545888869207400454). - GPT-3 can produce diagrams using Graphviz dot notation (https://twitter.com/goodside/status/1545109709035900928 https://twitter.com/goodside/status/1545109709035900928), including diagrams that incorporate real-world context (https://twitter.com/goodside/status/1545112039147847682 https://twitter.com/goodside/status/1545112039147847682) not given in the prompt. - It can emit and understand HTML and Markdown, including table markup — e.g., "Produce an HTML table giving the members of the Beatles and their instruments" - It can accept/emit multiple named files, with arbitrary content, simply by embedding those files as values in a larger JSON/YAML document. Here, (https://twitter.com/goodside/status/1545848754883919872 https://twitter.com/goodside/status/1545848754883919872) I demonstrate completing the file tree of a trivial Python package given a README.md. - It can understand the raw JSON file contents of .ipynb files, successfully following directions given to it in markdown cells, emitting multiple code cells, etc. While all this is fun, as many people have pointed out the model has a bad habit of "hallucinating" — producing highly specific, wrong output. Avoiding this (https://twitter.com/goodside/status/1545793388871651330 https://twitter.com/goodside/status/1545793388871651330) is more of an art than a science. Be highly skeptical of any output you get from GPT-3, and search for verification methods where you can.
- 323 4y agoHas anybody tried code reviews? As in "GPT-3, please do a code review of this code".
- goodside 4y agoFor small examples, yes, it works: Prompt: Perform a code review of the following module, identifying any bugs: ``` #!/usr/bin/env python3 def fib(n): """Return n-th Fibonacci number.""" if n <= 2: return 1 else: return fib(n-2) + fib(n+1) ``` GPT-3 completion: The bug in this module is that the n+1 parameter in the return statement should be n-1. But don't expect miracles on real-world files, because GPT-3 struggles with long-term dependencies in text (e.g., remembering what happened ~4000 characters ago).
- nephanth 4y agoNow I wonder what would happen if we finetuned GPT-3 on stackoverflow. Would it become better at code ? Good enough to assist any beginner in learning ? That would be huge
- goodside 4y agoThis is exactly the idea behind GPT-3 Codex, but using GitHub instead of SO. It does, indeed, become much better at programming than the baseline model.
- joshspankit 4y agoOne of the meta-challenges as I see it is “if we had a data set comprised of only the best information from the best experts we could make an AI that beats them all”. Because I think that’s both true and entirely unrealistic. Like, how would you quantify who the best experts are, and how would you verify that you’re not training on wrong data from those best experts? Maybe they had an off day or a specific agenda. Even browsing through verified SO answers there are still answers that are outdated and answers that are just plain wrong. The greater the experience needed to answer a specific question, the more unlikely it is that there is a clear and correct answer given.
- mmahemoff 4y agoSO is far from perfect, but of all the websites out there, SO has strong signal in this regard thanks to the moderation and metadata available.
- IshKebab 4y agoI'm sure the GPT-3 training set includes Stackoverflow already. Would be weird if it didn't.
- taneq 4y agoJust imagine being able to hit ctrl+space in your editor to add "this question has been marked as a duplicate." :P
- anewpersonality 4y agoHow the flying f*ck did they collect all the data for GPT-3?
- deleted 4y ago[deleted]
- tgv 4y agoThat work started back in the late 80s, early 90s, and people have been building and organizing corpora at an increasing scale. Cleaning a corpus is a hell of a lot of work. I guess they've been using older work to bootstrap that.
- IshKebab 4y agoScrape the web. Same way Google do it.
- kreetx 4y agoLooks like mostly scraped off the web https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilities https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilitie....
- bribri 4y agoI've been doing a lot of experiments with gpt-3 on twitter. Works with mermaid js https://twitter.com/Bsunter/status/1535062559069376512 https://twitter.com/Bsunter/status/1535062559069376512 Great for making study guides https://twitter.com/Bsunter/status/1535730704138444801 https://twitter.com/Bsunter/status/1535730704138444801 I also wrote a plugin to interact with gpt3 in the logseq text editor and some examples here. https://github.com/briansunter/logseq-plugin-gpt3-openai https://github.com/briansunter/logseq-plugin-gpt3-openai I have a few posts on my blog https://briansunter.com/blog/gtp3-openai-logseq-notetaking/ https://briansunter.com/blog/gtp3-openai-logseq-notetaking/ https://briansunter.com/graph/#/page/how%20to%20use%20gpt3%20to%20learn%20kubernetes https://briansunter.com/graph/#/page/how%20to%20use%20gpt3%2...
- _bramses 4y agoSelf-plug: I built Stenography(https://stenography.dev/ https://stenography.dev/) that does this for any codebase using smart parsing. The VSC code extension automatically writes code documentation
- goodside 4y agoThis looks cool, but I’d be more tempted to sign up if I could see examples of the documentation it can produce unassisted from an example codebase. My biggest concern is that the documentation would lack global context in a large file tree, and only be suitable for explaining small, self-contained files. The time investment to see if this is true or not is a bit too high as it is, so anything you can do to showcase the quality up-front might win you more signups.
- sureglymop 4y agoMan, you're cover of redbone made me want to try it out haha
- _bramses 4y agolmaooo, thank you XD
- la64710 4y agoThis is simply anazomg
- epakai 4y agoWould be interesting to produce changelogs or commit messages to be edited, but it would probably be abused more than it was helpful.
- bilsbie 4y agoI feel like we’re not appreciating how amazing this is.
- NO-AI 4y agoMany people here on Hacker News are realizing that a significant part of their hard-earned skills are now available to anyone for a few cents of compute time. Once these systems scale up to attention over long pieces of program text (instead of just a few thousand tokens) most of software development will be near minimum wage, almost skilled labor. You didn't mind when taxi drivers became obsolete. "I'm not a taxi driver, so I don't care!" We'll see how you feel when your economic value approaches zero.
- trention 4y agoDo you realize that writing the code is maybe ~25% of what a senior software engineer does?
- codedokode 4y agoThere are many tasks that do not require much intellect. For example, "here is a web store design mockup I have drawn. Generate the code for it. Integrate it with payment and delivery services". Today you need to hire developers, testers and other people. What if such trivial tasks could be performed by a neural network? Another example is phone support. AI is already being used there, for example, in some banks (in Russia) you are first greeted by a robot if you call, and only if it cannot answer your question it will let you talk to the human.
- trention 4y agoYour example is not at all as trivial as you may think when you delve into the details. And if you don't want to delve into details, there is already stuff like shopify.
- bilsbie 4y agoWe’re really in trouble when gpt can do stand ups.
- Datagenerator 4y agoHow can one use GPT3? ,Apt install GPT3' would be great
- goodside 4y agoYou need to give OpenAI your credit card and use their API. GPT-3 is not publicly available, and even if it were the hardware requirements for running it would be far, far beyond a typical desktop/server.
- simonw 4y agoYou can try it for free for three months. I wrote about how do to that here: https://simonwillison.net/2022/Jun/5/play-with-gpt3/ https://simonwillison.net/2022/Jun/5/play-with-gpt3/
- PoignardAzur 4y agoFWIW, various open language models are available on https://huggingface.co https://huggingface.co GPT3 isn't available; GPT2 is, as well as BLOOM which is (supposedly) as good. You can try them online, and you can download them. Downloading them isn't as easy as downloading as app, though: the models are pretty large and need some expertise and lots of compute to run. Though Hugging Face does offer to set you up with cloud providers. (I haven't actually try any of this, so I don't know how easy it is in practice. I've only tried the "playgrounds" that are available with some models)
- patrakov 4y agoCan it find and explain vulnerabilities such as SQL injections or buffer overflows?
- mjburgess 4y agoA NN is just a fancy search engine: the weights are a compression of its training data; it's prediction algorithm is a decompression+search. These "large models" are effective because the weights run to 100s GBs, so compressing TBs of data for searching. In any case, my question here is: how do you not go mad in this techno-religious environment where people (seemingly desperately and earnestly) equivocate correlations in written text with thinking? I feel at once overwhelmed, despairing, and a sense of need to point out that it isn't true. It's the most obscene form of wishful thinking and superstition. HNers seem capable of identifying pseudoscience when its an article in nutrition (namely, eg., that correlations in the effects of human behaviour are not explanations of it)... but loose all sense when "AI" becomes the buzzword. Correlations in *TEXT* are not models of language, intelligence, thinking. This is pseudoscience. And the most extreme and superstitious form: thinking that correlations are meaningful because we find them meaningful. A person using language uses it in response to an environment, so "Pass me the salt" is said on the occasion that one needs salt. The "language model", `language = person(...)` includes the environment as input (and many other things). Correlations in pixel patterns of the night sky are not a model of gravity. Correlations in text are not a model of language. Correlations in the detritus of human behaviour are not models of intelligence. But to my question: how does one cope in an environment of this kind of wishful thinking?
- anon2020dot00 4y agoWe know that correlation is not thinking, but it is good enough for it to help or to be used as a starting point which is why people appreciate it as code documentation in this case or code completion in Github co-pilot.
- mjburgess 4y agoSure, I mean I only explained my view so that the question would be taken seriously. For me the question is the reason I commented. I think I answered it myself... I have been seeing people's fervour to attribute intelligence to AI as either charlatanism or misunderstanding... but I think, now, it's actually wishful thinking (and hence essentially religious). Trading in the usual superstitious foundations of religions: "AI works in mysterious ways", "correlations in symbols are meaningful", etc. I still don't know why people wish that AI were intelligent; but at least now I have a clearer sense of it. There's some utopian faith behind it which creates this odd urgency to "require" machines to really be intelligent. I'd be interested in knowing why this wish exists. I can never quite put my finger on how irreligious tech people end up in these tech religions. I guess there's some attachment to there being "a heaven somewhere", which with tech, will be heaven-on-earth.
- _the_inflator 4y agoThis can make learning at lot more pleasant. Especially RegEx. Another area of huge interest is legacy code, or low level languages like assembler, COBOL, and some C stuff. Also almost everything without proper documentation. Well played.
- codeenlightener 4y agoWe have a product with a VS Code extension (IntelliJ and Chrome) coming soon that does this: https://denigma.app https://denigma.app
- ggktk 4y agoIt's very impressive what AI can do for developers to help us code. Imagine pair programming with an intern who memorized your database schema. You don't need to go back and forth between schema and your code, you can keep your focus at the cursor. It's great for automating the boring stuff. I would love if Github provided an API for Copilot. Imagine integrating Copilot with your shell.
- SLWW 4y agoI asked GPT-3 about several things ranging from creating for me a simple JS canvas (and embed it into it's own self-contained HTML doc) with a red bouncing ball. It interpreted that instruction to have the ball go up and down, starting at a diagonal and going from left to right, bouncing off all sides of the canvas. I then decided to ask it about Creator controversies which it gave surprisingly accurate and knowledgeable answers that pretty much mirror my own thoughts about a complex controversy that spanned over two years. It's kind of crazy, a little worrying with just how much the AI could gather from a few nasty reddit comment threads.
- lee101 4y agoHey also try out https://text-generator.io https://text-generator.io the same querying works but it will save you a load on gpt-3 cost https://text-generator.io/playground?text=SQLite+create+a+pages+table+that+has+a+compound+primary+key+of+type%2C+id+and+a+content+column.%0A%0ACREATE+TABLE&max_length=140 https://text-generator.io/playground?text=SQLite+create+a+pa... codex is technically free if you have access but soon it wont be so keep text-generator.io in mind there too as it can generate code from descriptions etc too at a wildly more competitive price. Had to use a bit of the "Create Table" syntax, but it did a create table if not exists which is nice... it missed the content field though
- make3 4y agocan't find the model size or a description of the dataset on the text-generator page
- jka 4y agoSlightly-curious, slightly-unnerved, slightly-paranoid question here: Is there any chance that this code snippet was sampled from repositories that contained associated commentary/discussion, guiding GPT-3 to produce similar explanations? Or is this genuinely an explanation that it can produce without context local to the code in question? And what number of people are able to determine the answer to my first question?
- hansword 4y ago>Is there any chance There is a chance. To get a better answer, you have to specify what exactly you mean by 'sampled' and by 'without context local to the code in question'.
- nprateem 4y agoIt may not be there today, but I think it's obvious that AI is going to basically make all creatives obsolete. At some point I expect it'll be able to: * Create movies, music and books on-demand * Design interiors, buildings, electric circuits, web sites, build pipelines, software architectures, etc. Software devs may be one of the last industries to be obsoleted, but it'll happen. About the only thing left will be the executive function of decision making, but even that's likely to devolve into: "AI, where's a gap in the market?" "Jonny, there are gaps in blah, blah, blah. Products that do x, y, z are in demand" "OK AI, design and print me a prototype" "Printing in progress..."
- Yhippa 4y agoI remember people online saying that automation and AI will free up our time to be creators instead of consumers. Looks like that's in jeopardy now. I guess back to consuming content for me. I feel like this could be one of the biggest technological changes I'll see in my lifetime but there isn't a lot of hype around it like say VR/AR and Web3. If this does end up exponentially improving, life is going to feel like magic.
- zbird 4y agoWill this ever be released? The fun stopped at step (1) Get an API token.
- deleted 4y ago[deleted]
- senthilnayagam 4y agothose who don't have GPT3 access and use GitHub Copilot, if you install copilot labs beta in VS code, you can get code explanation inside the editor itself, also bonus it offers language translation to other programming languages