4 ms·

The question answered by this page is "what is the first unused 3-letter acronym in English Wikipedia?" - it's CQK for the record. However, the meat of the page

The question answered by this page is "what is the first unused 3-letter acronym in English Wikipedia?" - it's CQK for the record. However, the meat of the page is how to effectively use GPT-4 to write this script, hence why I've submitted it under this title (go to https://gwern.net/tla#effective-gpt-4-programming https://gwern.net/tla#effective-gpt-4-programming). Interesting topics include: · Writing a good GPT-4 system prompt to make GPT-4 produce less verbose output and ask more questions. · How to iterate with GPT-4 to correct errors, generate a test suite, as well as a short design document (something you could put in the file-initial docstring in Python, for example). · The "blind spot" - if GPT-4 makes a subtle error with quoting, regex syntax, or similar, for example, it can be very tricky to tell GPT-4 how to correct the error, because it appears that it doesn't notice such errors very well, unlike higher-level errors. Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance. · If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script

staunton 3y ago
> Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance. By that argument, one should always make it use a language that's as hard as possible to write a compiling program. So Rust or Haskell or something? I guess at some point it's more important to have a lot of the language in the training data, too...
- smt88 3y ago
  I think this is exactly the right conclusion. The main complaint people have about strict, thorough type systems is that they have boilerplate. Obviously boilerplate doesn't matter if a machine writes the code. The type system also becomes helpful documentation of the intended behavior of the code that the LLM spits out.
  - kridsdale3 3y ago
    Assembly has a lot of boilerplate, and every other language is an abstraction that gets a language-machine to write it for us. So we'll just move to a new standard where we write LLM prompts describing function behavior and it will output the Rust or whatever that we end up storing in our SCM.
    - staunton 3y ago
      There's a fundamental difference though. The LLM is itself inscrutable, while all of these programs used to be written and understood by humans. The language used for programming used to be specified and have unique (hopefully) coherent syntax and abstraction boundaries. Now it's "anything goes" and nobody seems to know how this stuff ends up getting used... Someone might accidentally find it works well and then we might all end up writing fairytales in iambic pentameter describing the use cases of software we want...
- gwern 3y ago
  Yes, you would think so. Haskell would also be good for encouraging stateless/FP programming which makes unit-testing or property testing much easier. I can make GPT-4 write test-suites for functions which are straightforward data structure transformations, like rewriting strings, but I struggle to create tests for any of the imperative stuff. There presumably would be some way to test all of the imperative buffer editing Elisp code, but I have no idea what. However, in my use so far, I have not noticed any striking differences in error rates between Haskell and the others.
dang 3y ago
I modified the title slightly to use language from the subhead. (Submitted title was "Effective GPT-4 Programming", which does have the advantage of being a phrase from the article itself, but is more of a section heading than a description of the entire article. For the latter purpose, it's probably too generic.)
dtx1 3y ago
> If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script What an absolutely based take by GPT-4
- jondwillis 3y ago
  Really reflecting the bias of ML/AI practitioners to reach for a slow and footgunny language…
  - davrosthedalek 3y ago
    Slow yes, but footgunny? What do you find particularly footgunny about python?
JKCalhoun 3y ago
Where I grew up CQK was short for "Can't Quit the Koolaid." <jk>
- sebastiennight 3y ago
  Let's make it a thing! Does anybody have a UrbanDictionary account?

staunton 3y ago

> Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance. By that argument, one should always make it use a language that's as hard as possible to write a compiling program. So Rust or Haskell or something? I guess at some point it's more important to have a lot of the language in the training data, too...

smt88 3y ago
I think this is exactly the right conclusion. The main complaint people have about strict, thorough type systems is that they have boilerplate. Obviously boilerplate doesn't matter if a machine writes the code. The type system also becomes helpful documentation of the intended behavior of the code that the LLM spits out.
- kridsdale3 3y ago
  Assembly has a lot of boilerplate, and every other language is an abstraction that gets a language-machine to write it for us. So we'll just move to a new standard where we write LLM prompts describing function behavior and it will output the Rust or whatever that we end up storing in our SCM.
  - staunton 3y ago
    There's a fundamental difference though. The LLM is itself inscrutable, while all of these programs used to be written and understood by humans. The language used for programming used to be specified and have unique (hopefully) coherent syntax and abstraction boundaries. Now it's "anything goes" and nobody seems to know how this stuff ends up getting used... Someone might accidentally find it works well and then we might all end up writing fairytales in iambic pentameter describing the use cases of software we want...
gwern 3y ago
Yes, you would think so. Haskell would also be good for encouraging stateless/FP programming which makes unit-testing or property testing much easier. I can make GPT-4 write test-suites for functions which are straightforward data structure transformations, like rewriting strings, but I struggle to create tests for any of the imperative stuff. There presumably would be some way to test all of the imperative buffer editing Elisp code, but I have no idea what. However, in my use so far, I have not noticed any striking differences in error rates between Haskell and the others.

dang 3y ago

I modified the title slightly to use language from the subhead. (Submitted title was "Effective GPT-4 Programming", which does have the advantage of being a phrase from the article itself, but is more of a section heading than a description of the entire article. For the latter purpose, it's probably too generic.)

dtx1 3y ago

> If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script What an absolutely based take by GPT-4

jondwillis 3y ago
Really reflecting the bias of ML/AI practitioners to reach for a slow and footgunny language…
- davrosthedalek 3y ago
  Slow yes, but footgunny? What do you find particularly footgunny about python?

JKCalhoun 3y ago

Where I grew up CQK was short for "Can't Quit the Koolaid." <jk>

sebastiennight 3y ago
Let's make it a thing! Does anybody have a UrbanDictionary account?