6 ms·

For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output. Many of t

For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output. Many of these are discussed in the Twitter thread the OP links to, but I'd like to highlight a few of my favorites here too. - GPT-3 can produce simple SVG diagrams (https://twitter.com/goodside/status/1545883023719239689 https://twitter.com/goodside/status/1545883023719239689) from an English description, or incrementally modify a given SVG diagram (https://twitter.com/goodside/status/1545888869207400454 https://twitter.com/goodside/status/1545888869207400454). - GPT-3 can produce diagrams using Graphviz dot notation (https://twitter.com/goodside/status/1545109709035900928 https://twitter.com/goodside/status/1545109709035900928), including diagrams that incorporate real-world context (https://twitter.com/goodside/status/1545112039147847682 https://twitter.com/goodside/status/1545112039147847682) not given in the prompt. - It can emit and understand HTML and Markdown, including table markup — e.g., "Produce an HTML table giving the members of the Beatles and their instruments" - It can accept/emit multiple named files, with arbitrary content, simply by embedding those files as values in a larger JSON/YAML document. Here, (https://twitter.com/goodside/status/1545848754883919872 https://twitter.com/goodside/status/1545848754883919872) I demonstrate completing the file tree of a trivial Python package given a README.md. - It can understand the raw JSON file contents of .ipynb files, successfully following directions given to it in markdown cells, emitting multiple code cells, etc. While all this is fun, as many people have pointed out the model has a bad habit of "hallucinating" — producing highly specific, wrong output. Avoiding this (https://twitter.com/goodside/status/1545793388871651330 https://twitter.com/goodside/status/1545793388871651330) is more of an art than a science. Be highly skeptical of any output you get from GPT-3, and search for verification methods where you can.

323 4y ago
Has anybody tried code reviews? As in "GPT-3, please do a code review of this code".
- goodside 4y ago
  For small examples, yes, it works: Prompt: Perform a code review of the following module, identifying any bugs: ``` #!/usr/bin/env python3 def fib(n): """Return n-th Fibonacci number.""" if n <= 2: return 1 else: return fib(n-2) + fib(n+1) ``` GPT-3 completion: The bug in this module is that the n+1 parameter in the return statement should be n-1. But don't expect miracles on real-world files, because GPT-3 struggles with long-term dependencies in text (e.g., remembering what happened ~4000 characters ago).
  - zitterbewegung 4y ago
    Seems to work for a factorial function also Perform a code review of the following module, identifying any bugs: ``` #!/usr/bin/env python3 def factorial(x): """This is a recursive function to find the factorial of an integer""" if x == 1: return 1 else: # recursive call to the function return (x * factorial(x+1)) ``` A: The bug in this code is that the function is attempting to find the factorial of an integer by recursively calling itself with an incremented value. This will cause an infinite loop. But if you unindent and then also perform the n+1 error it only will see the unindent issue. Perform a code review of the following module, identifying any bugs: ``` #!/usr/bin/env python3 def factorial(x): """This is a recursive function to find the factorial of an integer""" if x == 1: return 1 else: # recursive call to the function return (x * factorial(x+1)) ``` A: The code will not run because there is a syntax error on line 6. The indentation is incorrect. The return statement should not be indented.
  - deleted 4y ago
    [deleted]
- jggggggggggggj 4y ago
6keZbCECT2uB 4y ago
This is very cool. Two uses come to mind: 1. Alternative variable name suggestions. I am writing code, and I try to figure out what to name a variable that about a data frame which is my performance data which takes the maximum of random samples and batch size leaving data size and thread count, and all I can come up with is unbatched_speed or something. I can think of a lot of scenarios where a context sensitive thesaurus / autocomplete would help. If it's obvious to an AI, maybe it'll be obvious to a junior dev? 2. I expect there are a lot of cases where not technical people could understand code, but get intimidated because it looks like code. An AI to say what the code for the business logic that they care about, but using prose.
- goodside 4y ago
  #1 is feasible now for short (<4K chars or so) blocks of code. Just paste the code in an indented block as you would in a comment, then after the block ask in English for a list of suggestions for what the variable could be renamed to.
- bilsbie 4y ago
  2. Is a great idea! I bet that could be useful. Maybe a way to do the same with formulas. I always get hung up on the odd notation and the symbols that aren’t defined anywhere.
- bobbylarrybobby 4y ago
  2 sounds like auto-generated long-form documentation? That would be a major win for devs and non-devs alike.
  - simonw 4y ago
    I tried writing a tutorial with GPT-3 a few months ago with quite impressive results: https://simonwillison.net/2022/May/31/a-datasette-tutorial-written-by-gpt-3/ https://simonwillison.net/2022/May/31/a-datasette-tutorial-w...
pmoriarty 4y ago
"For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output" I'd like to see it get good at continuing poetry.. It's pretty awful at that, from my own experiments.
- goodside 4y ago
  I enjoyed this one enough that I’ve committed the sonnet to memory: https://mobile.twitter.com/xkcd/status/1513922269122281473 https://mobile.twitter.com/xkcd/status/1513922269122281473
- ppod 4y ago
  It seems to understand a fair bit about themes and rhythm but it just cannot do rhyme. Something to do with the token encoding.

323 4y ago

Has anybody tried code reviews? As in "GPT-3, please do a code review of this code".

goodside 4y ago
For small examples, yes, it works: Prompt: Perform a code review of the following module, identifying any bugs: ``` #!/usr/bin/env python3 def fib(n): """Return n-th Fibonacci number.""" if n <= 2: return 1 else: return fib(n-2) + fib(n+1) ``` GPT-3 completion: The bug in this module is that the n+1 parameter in the return statement should be n-1. But don't expect miracles on real-world files, because GPT-3 struggles with long-term dependencies in text (e.g., remembering what happened ~4000 characters ago).
- zitterbewegung 4y ago
  Seems to work for a factorial function also Perform a code review of the following module, identifying any bugs: ``` #!/usr/bin/env python3 def factorial(x): """This is a recursive function to find the factorial of an integer""" if x == 1: return 1 else: # recursive call to the function return (x * factorial(x+1)) ``` A: The bug in this code is that the function is attempting to find the factorial of an integer by recursively calling itself with an incremented value. This will cause an infinite loop. But if you unindent and then also perform the n+1 error it only will see the unindent issue. Perform a code review of the following module, identifying any bugs: ``` #!/usr/bin/env python3 def factorial(x): """This is a recursive function to find the factorial of an integer""" if x == 1: return 1 else: # recursive call to the function return (x * factorial(x+1)) ``` A: The code will not run because there is a syntax error on line 6. The indentation is incorrect. The return statement should not be indented.
- deleted 4y ago
  [deleted]
jggggggggggggj 4y ago

6keZbCECT2uB 4y ago

This is very cool. Two uses come to mind: 1. Alternative variable name suggestions. I am writing code, and I try to figure out what to name a variable that about a data frame which is my performance data which takes the maximum of random samples and batch size leaving data size and thread count, and all I can come up with is unbatched_speed or something. I can think of a lot of scenarios where a context sensitive thesaurus / autocomplete would help. If it's obvious to an AI, maybe it'll be obvious to a junior dev? 2. I expect there are a lot of cases where not technical people could understand code, but get intimidated because it looks like code. An AI to say what the code for the business logic that they care about, but using prose.

goodside 4y ago
#1 is feasible now for short (<4K chars or so) blocks of code. Just paste the code in an indented block as you would in a comment, then after the block ask in English for a list of suggestions for what the variable could be renamed to.
bilsbie 4y ago
2. Is a great idea! I bet that could be useful. Maybe a way to do the same with formulas. I always get hung up on the odd notation and the symbols that aren’t defined anywhere.
bobbylarrybobby 4y ago
2 sounds like auto-generated long-form documentation? That would be a major win for devs and non-devs alike.
- simonw 4y ago
  I tried writing a tutorial with GPT-3 a few months ago with quite impressive results: https://simonwillison.net/2022/May/31/a-datasette-tutorial-written-by-gpt-3/ https://simonwillison.net/2022/May/31/a-datasette-tutorial-w...

pmoriarty 4y ago

"For the past few weeks, I've been collecting and refining prompting tricks for getting GPT-3 to emit and understand various types of non-prose output" I'd like to see it get good at continuing poetry.. It's pretty awful at that, from my own experiments.

goodside 4y ago
I enjoyed this one enough that I’ve committed the sonnet to memory: https://mobile.twitter.com/xkcd/status/1513922269122281473 https://mobile.twitter.com/xkcd/status/1513922269122281473
ppod 4y ago
It seems to understand a fair bit about themes and rhythm but it just cannot do rhyme. Something to do with the token encoding.