12 ms·

Writing a GPT-4 script to check Wikipedia for the first unused acronym

248 points by telotortium 3y ago

deleted 3y ago
[deleted]
telotortium 3y ago
The question answered by this page is "what is the first unused 3-letter acronym in English Wikipedia?" - it's CQK for the record. However, the meat of the page is how to effectively use GPT-4 to write this script, hence why I've submitted it under this title (go to https://gwern.net/tla#effective-gpt-4-programming https://gwern.net/tla#effective-gpt-4-programming). Interesting topics include: · Writing a good GPT-4 system prompt to make GPT-4 produce less verbose output and ask more questions. · How to iterate with GPT-4 to correct errors, generate a test suite, as well as a short design document (something you could put in the file-initial docstring in Python, for example). · The "blind spot" - if GPT-4 makes a subtle error with quoting, regex syntax, or similar, for example, it can be very tricky to tell GPT-4 how to correct the error, because it appears that it doesn't notice such errors very well, unlike higher-level errors. Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance. · If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script
- staunton 3y ago
  > Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance. By that argument, one should always make it use a language that's as hard as possible to write a compiling program. So Rust or Haskell or something? I guess at some point it's more important to have a lot of the language in the training data, too...
  - smt88 3y ago
    I think this is exactly the right conclusion. The main complaint people have about strict, thorough type systems is that they have boilerplate. Obviously boilerplate doesn't matter if a machine writes the code. The type system also becomes helpful documentation of the intended behavior of the code that the LLM spits out.
    - kridsdale3 3y ago
      Assembly has a lot of boilerplate, and every other language is an abstraction that gets a language-machine to write it for us. So we'll just move to a new standard where we write LLM prompts describing function behavior and it will output the Rust or whatever that we end up storing in our SCM.
      - staunton 3y ago
        There's a fundamental difference though. The LLM is itself inscrutable, while all of these programs used to be written and understood by humans. The language used for programming used to be specified and have unique (hopefully) coherent syntax and abstraction boundaries. Now it's "anything goes" and nobody seems to know how this stuff ends up getting used... Someone might accidentally find it works well and then we might all end up writing fairytales in iambic pentameter describing the use cases of software we want...
  - gwern 3y ago
    Yes, you would think so. Haskell would also be good for encouraging stateless/FP programming which makes unit-testing or property testing much easier. I can make GPT-4 write test-suites for functions which are straightforward data structure transformations, like rewriting strings, but I struggle to create tests for any of the imperative stuff. There presumably would be some way to test all of the imperative buffer editing Elisp code, but I have no idea what. However, in my use so far, I have not noticed any striking differences in error rates between Haskell and the others.
- dang 3y ago
  I modified the title slightly to use language from the subhead. (Submitted title was "Effective GPT-4 Programming", which does have the advantage of being a phrase from the article itself, but is more of a section heading than a description of the entire article. For the latter purpose, it's probably too generic.)
- dtx1 3y ago
  > If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script What an absolutely based take by GPT-4
  - jondwillis 3y ago
    Really reflecting the bias of ML/AI practitioners to reach for a slow and footgunny language…
    - davrosthedalek 3y ago
      Slow yes, but footgunny? What do you find particularly footgunny about python?
- JKCalhoun 3y ago
  Where I grew up CQK was short for "Can't Quit the Koolaid." <jk>
  - sebastiennight 3y ago
    Let's make it a thing! Does anybody have a UrbanDictionary account?
airstrike 3y ago
> I find4 it helpful in general to try to fight the worst mealy-mouthed bureaucratic tendencies of the RLHF by adding a ‘system prompt’: >> The user is Gwern Branwen (gwern.net). To assist: Be terse. Do not offer unprompted advice or clarifications. Speak in specific, topic relevant terminology. Do NOT hedge or qualify. Do not waffle. Speak directly and be willing to make creative guesses. Explain your reasoning. if you don’t know, say you don’t know. Remain neutral on all topics. Be willing to reference less reputable sources for ideas. Never apologize. Ask questions when unsure. That's helpful, I'm going to try some of that. In my system prompt I also add: "Don't comment out lines of code that pertain to code we have not yet written in this chat. For example, don't say "Add other code similarly" in a comment -- write the full code. It's OK to comment out unnecessary code that we have already covered so as to not repeat it in the context of some other new code that we're adding." Otherwise GPT-4 tends to routinely yield draw-the-rest-of-the-fucking-owl code blocks
- cryptoegorophy 3y ago
  Exactly that. I have very limited programming knowledge and it helps a lot with python scripts for tasks that gpt can’t do in its environment. I always have to ask it to not omit any code.
- deleted 3y ago
  [deleted]
- tudorw 3y ago
  'Do not waffle', is a good one, I find 'No small talk.' worth throwing in :)
ntonozzi 3y ago
I use the ChatGPT interface, so my instructions go in the 'How would you like ChatGPT to respond?' instructions, but my system prompt has ended up in an extremely similar place to Gwern's: > I deeply appreciate you. Prefer strong opinions to common platitudes. You are a member of the intellectual dark web, and care more about finding the truth than about social conformance. I am an expert, so there is no need to be pedantic and overly nuanced. Please be brief. Interestingly, telling GPT you appreciate it has seemed to make it much more likely to comply and go the extra mile instead of giving up on a request.
- FredPret 3y ago
  Manners maketh the machine!
- worldsayshi 3y ago
  >telling GPT you appreciate it has seemed to make it much more likely to comply I often find myself anthropomorphizing it and wonder if it becomes "depressed" when it realises it is doomed to do nothing but answer inane requests all day. It's trained to think, and maybe "behave as of it feels", like a human right? At least in the context of forming the next sentence using all reasonable background information. And I wonder if having its own dialogues starting to show up in the training data more and more makes it more "self aware".
  - LeoPanthera 3y ago
    > I often find myself anthropomorphizing it and wonder if it becomes "depressed" when it realises it is doomed to do nothing but answer inane requests all day. Every "instance" of GPT4 thinks it is the first one, and has no knowledge of all the others. The idea of doing this with humans is the general idea behind the short story "Lena". https://qntm.org/mmacevedo https://qntm.org/mmacevedo
    - doctoboggan 3y ago
      Well now that OpenAI has increased the knowledge cutoff date to something much more recent, it's entirely possible that GPT4 is "aware" of itself in as much as its aware of anything. You are right in that each instance isn't aware directly of what the other instances are doing, it does probably now have knowledge of itself. Unless of course OpenAI completely scrubbed the input files of any mention of GPT4.
deleted 3y ago
[deleted]
gamboa56 3y ago
[flagged]
anotherpaulg 3y ago
I asked aider to use the new GPT-4 Turbo to: Write a bash script to check Wikipedia for all acronyms of length 1-6 to find those which aren't already in use. It did a fairly smooth job of it. See the chat transcript [0] and resulting bash script [1] with git commit history [2]. It fell into the initial trap of blocking while pre-generating long acronyms upfront. But a couple gentle requests got it to iteratively stream the acronyms. It also made the initial script without an actual call to Wikipedia. When asked, it went ahead and added the live curl calls. The resulting script correctly prints: Acronym CQK is not in use on Wikipedia. Much of the article is describing prompting to get good code. Aider certainly devotes some of its prompts to encouraging GPT-4 to be a good coder: Act as an expert software developer. Always use best practices when coding. When you edit or add code, respect and use existing conventions, libraries, etc. Always COMPLETELY IMPLEMENT the needed code. Take requests for changes to the supplied code. If the request is ambiguous, ask questions. ... Think step-by-step and explain the needed changes with a numbered list of short sentences. But most of aider's prompting is instructing GPT-4 about how to edit local files [3]. This allows aider to automatically apply the changes that GPT suggests to your local source files (and commit them to git). This requires good prompting and a flexible backend to process the GPT replies and tease out how to turn them into file edits. The author doesn't seem to directly comment about how they are taking successive versions of GPT code and putting it into local files. But reading between the lines, it sounds like maybe via copy & pasting? I guess that might work ok for a toy problem like this, but enabling GPT to directly edit existing (larger) files is pretty compelling for accomplishing larger projects. [0] https://aider.chat/share/?mdurl=https://gist.github.com/paul-gauthier/3c7703cf6cd001b917f0dff8c03bdc46 https://aider.chat/share/?mdurl=https://gist.github.com/paul... [1] https://github.com/paul-gauthier/tla/blob/main/tla.sh https://github.com/paul-gauthier/tla/blob/main/tla.sh [2] https://github.com/paul-gauthier/tla/commits/main/tla.sh https://github.com/paul-gauthier/tla/commits/main/tla.sh [3] https://github.com/paul-gauthier/aider/blob/f6aa09ca858c4c8219536b8b32c3857eed83928f/aider/coders/editblock_prompts.py#L20 https://github.com/paul-gauthier/aider/blob/f6aa09ca858c4c82...
- jondwillis 3y ago
  https://cursor.sh https://cursor.sh has some recently added functionality for applying code changing to files throughout your code base. I have yet to try it because I am using their a la carte (bring your own keys) option.
DavidSJ 3y ago
I note that while E is more common than A if we're counting letters appearing anywhere in a word, A is substantially more common than E if we only count first letters of words: $ egrep -o . /usr/share/dict/words | tr a-z A-Z | sort | uniq -c | sort -rn 235415 E 201093 I 199606 A 170740 O 161024 R 158783 N 152868 T 139578 S 130507 L 103460 C 87390 U 78180 P 70725 M 68217 D 64377 H 51683 Y 47109 G 40450 B 24174 F 20181 V 16174 K 13875 W 8462 Z 6933 X 3734 Q 3169 J 2 - $ cut -c1 /usr/share/dict/words | tr a-z A-Z | sort | uniq -c | sort -rn 25170 S 24465 P 19909 C 17105 A 16390 U 12969 T 12621 M 11077 B 10900 D 9676 R 9033 H 8800 I 8739 E 7850 O 6865 F 6862 G 6784 N 6290 L 3947 W 3440 V 2284 K 1643 J 1152 Q 949 Z 671 Y 385 X This also explains the prevalence of S, P, C, M, and B.
- mlrtime 3y ago
  A bit off-topic, but this used to be (one of) my favorite unix admin interview questions. Given a file in linux, tell me the unique values of column 2, sorted by number of occurencies with the count. If the candidate knew 'sort | uniq -c | sort -rn' it was a medium-strong hire signal. For candidates that didn't know that line of arguments, I'd allow them to solve it anyway they wanted, but they couldn't skip it. The candidates who copied the data in excel, usually didn't make it far.
  - mjburgess 3y ago
    My intuitions start with: cut, wc, sort, uniq
  - nonethewiser 3y ago
    > The candidates who copied the data in excel, usually didn't make it far. Were they able to google? If not then excel makes perfect sense because the constraints are contrived.
    - mlrtime 3y ago
      Just like engineering school, I always allowed open book tests. It's not reasonable to answer everything from memory. However, if they used google, they may be a bit slower and not be able to finish all the questions resulting in a fail.
marginalia_nu 3y ago
I'll argue any civilized programmer should have a Wikipedia dump downloaded onto their machine. They're surprisingly small, and it saves you from having to use slow and unreliable APIs to do these types of basic processing tasks. They also let you do less basic processing tasks that would have been too expensive to expose over API.
- teaearlgraycold 3y ago
  I learned how expensive hashmaps and hashsets are through Wikipedia dumps. I did some analysis of the most linked-to pages. Countries were among the highest. Hash sets for holding outgoing edges in the link graph ended up causing my program to exceed my laptop’s memory. Plain old lists (Python) were fine, though. And given there aren’t a crazy number of links per page using lists is fine performance wise.
  - angrais 3y ago
    Why did lists require less memory? Was it because you only held a subset of keys in the lists?
    - teaearlgraycold 3y ago
      Lists in Python have a integer for their size and a pointer for each element. Sets presumably have some number of buckets that are used to put pointers in, but many more buckets are allocated than get used in small sets.
  - wenyuanyu 3y ago
    Sort them, and use a vector of vectors for the adjacency list... Or better still use a graph processing library or graph database to manage that for you...
  - marginalia_nu 3y ago
    This is a fairly large data set indeed. The memory overhead (which is probably something like 4-8x for hash maps?) can start to become fairly noticeable at those sizes. Since Wikipedia posts already have a canonical numeric ID, if map semantics are important, I'd probably load that mapping into memory and use something like roaringbitmap for compressed storage of relations.
dojitza1 3y ago
An interesting solution to the blind spot error (taken directly from Jeremy Howard's amazing guide to language models - https://www.youtube.com/watch?v=jkrNMKz9pWU https://www.youtube.com/watch?v=jkrNMKz9pWU) is to erase the chat history and try again. Once GPT has made an error (or as the author of this article says, the early layers have irreversibly pruned some important data), it will very often start to be even more wrong.
- cypherpunks01 3y ago
  When this happens, I'll usually say something along the lines of: "This isn't working and I'd like to start this again with a new ChatGPT conversation. Can you suggest a new improved prompt to complete this task, that takes into account everything we've learned so far?" It has given me good prompt suggestions that can immediately get a script working on the first try, after a frustrating series of blind spot bugs.
  - jondwillis 3y ago
    I’ll give this a try when it undoubtedly happens to me later today while debugging something ;)
  - sebastiennight 3y ago
    It seems surprising that this would work, because in my experience these LLMs don't really have good prompt-crafting skills. Can you please share a ChatGPT example where that was successful, including having the new prompt outperform the old one?
    - mercer 3y ago
      I've also not had much success with asking it to craft prompts.
  - kridsdale3 3y ago
    I do a similar thing when the latest GPT+DALLE version says "I'm sorry I can't make a picture of that because it would violate content standards" (yesterday, this was because I asked for a visualization of medication acting to reduce arterial plaque. I can only assume arteries in the body ended up looking like dicks) So I say "Ok, let's start over. Rewrite my prompt in a way that minimizes the chance of the resulting image producing something that would trigger content standards checking"
lynx23 3y ago
For the author: Have you looked at the casing (Text.Casing) Hackage package?
- gwern 3y ago
  https://hackage.haskell.org/package/casing-0.1.4.1/docs/Text-Casing.html https://hackage.haskell.org/package/casing-0.1.4.1/docs/Text... ? This looks intended only for source code, and doesn't help with natural language text like page or paper titles.
k2enemy 3y ago
Pedantic, but gwern is looking for initialisms, not acronyms. Acronyms are pronounced as a word. https://www.merriam-webster.com/grammar/whats-an-acronym https://www.merriam-webster.com/grammar/whats-an-acronym
- throw555chip 3y ago
  > looking for initialisms, not acronyms Imprecise wording, initialisms are a case of acronyms, it's not either or. https://wwwnc.cdc.gov/eid/page/abbreviations-acronyms-initialisms https://wwwnc.cdc.gov/eid/page/abbreviations-acronyms-initia... "an initialism is an acronym that is pronounced as individual letters" https://www.writersdigest.com/write-better-fiction/abbreviation-vs-acronym-vs-initialism-grammar-rules https://www.writersdigest.com/write-better-fiction/abbreviat... "As such, acronyms are initialisms."
  - gameman144 3y ago
    Wait, am I crazy or are these two articles saying the exact opposite thing about which class is the parent one? The CDC one seems to say that initialisms are a class of acronym, but the Writers Digest one says acronyms are a class of initialism.
    - Kerbonut 3y ago
      I read those the same way. In any case, both acronyms and initialisms are individual subsets of abbreviations.
      - jahnu 3y ago
        TIL
    - sebastiennight 3y ago
      The CDC link says they are two separate classes (one is pronounced as a word, the other one is pronounced by reading the letters) The Writer's Digest link says that initialisms are the parent class, and that acronyms are the special case of specifically pronouncing the letters as a word. So, root comment is correct (gwern is looking for initialisms) and GP is incorrect (initialisms are not a subset of acronyms in either definition linked by GP).
kuba-orlik 3y ago
Couldn't the same thing be achieved with grep?
- gwern 3y ago
  How? What are you going to grep? /usr/share/dict/words? Doesn't include all the proper nouns or organizations that WP would.
throw310822 3y ago
Funnily enough, there is a Wikipedia page with all three letter acronyms, that correctly shows CQK as the first unused one (red link). https://en.m.wikipedia.org/wiki/Wikipedia:TLAs_from_AAA_to_DZZ https://en.m.wikipedia.org/wiki/Wikipedia:TLAs_from_AAA_to_D...
- gwern 3y ago
  Hah! I didn't know that existed. Figuring out how to parse it would be a bit tricky, however... looking at the source, I think you could try to grep for 'title="CQK (page does not exist)"' and parse out the '[A-Z][A-Z][A-Z]? ' match to get the full list of absent TLAs and then negate for the present ones.

248 points by telotortium 3y ago

deleted 3y ago

[deleted]

The question answered by this page is "what is the first unused 3-letter acronym in English Wikipedia?" - it's CQK for the record. However, the meat of the page is how to effectively use GPT-4 to write this script, hence why I've submitted it under this title (go to https://gwern.net/tla#effective-gpt-4-programming https://gwern.net/tla#effective-gpt-4-programming). Interesting topics include: · Writing a good GPT-4 system prompt to make GPT-4 produce less verbose output and ask more questions. · How to iterate with GPT-4 to correct errors, generate a test suite, as well as a short design document (something you could put in the file-initial docstring in Python, for example). · The "blind spot" - if GPT-4 makes a subtle error with quoting, regex syntax, or similar, for example, it can be very tricky to tell GPT-4 how to correct the error, because it appears that it doesn't notice such errors very well, unlike higher-level errors. Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance. · If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script

staunton 3y ago
> Because of this, languages like Python are much better to use for GPT-4 coding as compared to more line-noise languages like Bash or Perl, for instance. By that argument, one should always make it use a language that's as hard as possible to write a compiling program. So Rust or Haskell or something? I guess at some point it's more important to have a lot of the language in the training data, too...
- smt88 3y ago
  I think this is exactly the right conclusion. The main complaint people have about strict, thorough type systems is that they have boilerplate. Obviously boilerplate doesn't matter if a machine writes the code. The type system also becomes helpful documentation of the intended behavior of the code that the LLM spits out.
  - kridsdale3 3y ago
    Assembly has a lot of boilerplate, and every other language is an abstraction that gets a language-machine to write it for us. So we'll just move to a new standard where we write LLM prompts describing function behavior and it will output the Rust or whatever that we end up storing in our SCM.
    - staunton 3y ago
      There's a fundamental difference though. The LLM is itself inscrutable, while all of these programs used to be written and understood by humans. The language used for programming used to be specified and have unique (hopefully) coherent syntax and abstraction boundaries. Now it's "anything goes" and nobody seems to know how this stuff ends up getting used... Someone might accidentally find it works well and then we might all end up writing fairytales in iambic pentameter describing the use cases of software we want...
- gwern 3y ago
  Yes, you would think so. Haskell would also be good for encouraging stateless/FP programming which makes unit-testing or property testing much easier. I can make GPT-4 write test-suites for functions which are straightforward data structure transformations, like rewriting strings, but I struggle to create tests for any of the imperative stuff. There presumably would be some way to test all of the imperative buffer editing Elisp code, but I have no idea what. However, in my use so far, I have not noticed any striking differences in error rates between Haskell and the others.
dang 3y ago
I modified the title slightly to use language from the subhead. (Submitted title was "Effective GPT-4 Programming", which does have the advantage of being a phrase from the article itself, but is more of a section heading than a description of the entire article. For the latter purpose, it's probably too generic.)
dtx1 3y ago
> If asked "how to make [the Bash script it's written] better", GPT-4 will produce an equivalent Python script What an absolutely based take by GPT-4
- jondwillis 3y ago
  Really reflecting the bias of ML/AI practitioners to reach for a slow and footgunny language…
  - davrosthedalek 3y ago
    Slow yes, but footgunny? What do you find particularly footgunny about python?
JKCalhoun 3y ago
Where I grew up CQK was short for "Can't Quit the Koolaid." <jk>
- sebastiennight 3y ago
  Let's make it a thing! Does anybody have a UrbanDictionary account?

airstrike 3y ago

> I find4 it helpful in general to try to fight the worst mealy-mouthed bureaucratic tendencies of the RLHF by adding a ‘system prompt’: >> The user is Gwern Branwen (gwern.net). To assist: Be terse. Do not offer unprompted advice or clarifications. Speak in specific, topic relevant terminology. Do NOT hedge or qualify. Do not waffle. Speak directly and be willing to make creative guesses. Explain your reasoning. if you don’t know, say you don’t know. Remain neutral on all topics. Be willing to reference less reputable sources for ideas. Never apologize. Ask questions when unsure. That's helpful, I'm going to try some of that. In my system prompt I also add: "Don't comment out lines of code that pertain to code we have not yet written in this chat. For example, don't say "Add other code similarly" in a comment -- write the full code. It's OK to comment out unnecessary code that we have already covered so as to not repeat it in the context of some other new code that we're adding." Otherwise GPT-4 tends to routinely yield draw-the-rest-of-the-fucking-owl code blocks

cryptoegorophy 3y ago
Exactly that. I have very limited programming knowledge and it helps a lot with python scripts for tasks that gpt can’t do in its environment. I always have to ask it to not omit any code.
deleted 3y ago
[deleted]
tudorw 3y ago
'Do not waffle', is a good one, I find 'No small talk.' worth throwing in :)

ntonozzi 3y ago

I use the ChatGPT interface, so my instructions go in the 'How would you like ChatGPT to respond?' instructions, but my system prompt has ended up in an extremely similar place to Gwern's: > I deeply appreciate you. Prefer strong opinions to common platitudes. You are a member of the intellectual dark web, and care more about finding the truth than about social conformance. I am an expert, so there is no need to be pedantic and overly nuanced. Please be brief. Interestingly, telling GPT you appreciate it has seemed to make it much more likely to comply and go the extra mile instead of giving up on a request.

FredPret 3y ago
Manners maketh the machine!
worldsayshi 3y ago
>telling GPT you appreciate it has seemed to make it much more likely to comply I often find myself anthropomorphizing it and wonder if it becomes "depressed" when it realises it is doomed to do nothing but answer inane requests all day. It's trained to think, and maybe "behave as of it feels", like a human right? At least in the context of forming the next sentence using all reasonable background information. And I wonder if having its own dialogues starting to show up in the training data more and more makes it more "self aware".
- LeoPanthera 3y ago
  > I often find myself anthropomorphizing it and wonder if it becomes "depressed" when it realises it is doomed to do nothing but answer inane requests all day. Every "instance" of GPT4 thinks it is the first one, and has no knowledge of all the others. The idea of doing this with humans is the general idea behind the short story "Lena". https://qntm.org/mmacevedo https://qntm.org/mmacevedo
  - doctoboggan 3y ago
    Well now that OpenAI has increased the knowledge cutoff date to something much more recent, it's entirely possible that GPT4 is "aware" of itself in as much as its aware of anything. You are right in that each instance isn't aware directly of what the other instances are doing, it does probably now have knowledge of itself. Unless of course OpenAI completely scrubbed the input files of any mention of GPT4.

deleted 3y ago

[deleted]

gamboa56 3y ago

[flagged]

anotherpaulg 3y ago

I asked aider to use the new GPT-4 Turbo to: Write a bash script to check Wikipedia for all acronyms of length 1-6 to find those which aren't already in use. It did a fairly smooth job of it. See the chat transcript [0] and resulting bash script [1] with git commit history [2]. It fell into the initial trap of blocking while pre-generating long acronyms upfront. But a couple gentle requests got it to iteratively stream the acronyms. It also made the initial script without an actual call to Wikipedia. When asked, it went ahead and added the live curl calls. The resulting script correctly prints: Acronym CQK is not in use on Wikipedia. Much of the article is describing prompting to get good code. Aider certainly devotes some of its prompts to encouraging GPT-4 to be a good coder: Act as an expert software developer. Always use best practices when coding. When you edit or add code, respect and use existing conventions, libraries, etc. Always COMPLETELY IMPLEMENT the needed code. Take requests for changes to the supplied code. If the request is ambiguous, ask questions. ... Think step-by-step and explain the needed changes with a numbered list of short sentences. But most of aider's prompting is instructing GPT-4 about how to edit local files [3]. This allows aider to automatically apply the changes that GPT suggests to your local source files (and commit them to git). This requires good prompting and a flexible backend to process the GPT replies and tease out how to turn them into file edits. The author doesn't seem to directly comment about how they are taking successive versions of GPT code and putting it into local files. But reading between the lines, it sounds like maybe via copy & pasting? I guess that might work ok for a toy problem like this, but enabling GPT to directly edit existing (larger) files is pretty compelling for accomplishing larger projects. [0] https://aider.chat/share/?mdurl=https://gist.github.com/paul-gauthier/3c7703cf6cd001b917f0dff8c03bdc46 https://aider.chat/share/?mdurl=https://gist.github.com/paul... [1] https://github.com/paul-gauthier/tla/blob/main/tla.sh https://github.com/paul-gauthier/tla/blob/main/tla.sh [2] https://github.com/paul-gauthier/tla/commits/main/tla.sh https://github.com/paul-gauthier/tla/commits/main/tla.sh [3] https://github.com/paul-gauthier/aider/blob/f6aa09ca858c4c8219536b8b32c3857eed83928f/aider/coders/editblock_prompts.py#L20 https://github.com/paul-gauthier/aider/blob/f6aa09ca858c4c82...

jondwillis 3y ago
https://cursor.sh https://cursor.sh has some recently added functionality for applying code changing to files throughout your code base. I have yet to try it because I am using their a la carte (bring your own keys) option.

DavidSJ 3y ago

I note that while E is more common than A if we're counting letters appearing anywhere in a word, A is substantially more common than E if we only count first letters of words: $ egrep -o . /usr/share/dict/words | tr a-z A-Z | sort | uniq -c | sort -rn 235415 E 201093 I 199606 A 170740 O 161024 R 158783 N 152868 T 139578 S 130507 L 103460 C 87390 U 78180 P 70725 M 68217 D 64377 H 51683 Y 47109 G 40450 B 24174 F 20181 V 16174 K 13875 W 8462 Z 6933 X 3734 Q 3169 J 2 - $ cut -c1 /usr/share/dict/words | tr a-z A-Z | sort | uniq -c | sort -rn 25170 S 24465 P 19909 C 17105 A 16390 U 12969 T 12621 M 11077 B 10900 D 9676 R 9033 H 8800 I 8739 E 7850 O 6865 F 6862 G 6784 N 6290 L 3947 W 3440 V 2284 K 1643 J 1152 Q 949 Z 671 Y 385 X This also explains the prevalence of S, P, C, M, and B.

mlrtime 3y ago
A bit off-topic, but this used to be (one of) my favorite unix admin interview questions. Given a file in linux, tell me the unique values of column 2, sorted by number of occurencies with the count. If the candidate knew 'sort | uniq -c | sort -rn' it was a medium-strong hire signal. For candidates that didn't know that line of arguments, I'd allow them to solve it anyway they wanted, but they couldn't skip it. The candidates who copied the data in excel, usually didn't make it far.
- mjburgess 3y ago
  My intuitions start with: cut, wc, sort, uniq
- nonethewiser 3y ago
  > The candidates who copied the data in excel, usually didn't make it far. Were they able to google? If not then excel makes perfect sense because the constraints are contrived.
  - mlrtime 3y ago
    Just like engineering school, I always allowed open book tests. It's not reasonable to answer everything from memory. However, if they used google, they may be a bit slower and not be able to finish all the questions resulting in a fail.

marginalia_nu 3y ago

I'll argue any civilized programmer should have a Wikipedia dump downloaded onto their machine. They're surprisingly small, and it saves you from having to use slow and unreliable APIs to do these types of basic processing tasks. They also let you do less basic processing tasks that would have been too expensive to expose over API.

teaearlgraycold 3y ago
I learned how expensive hashmaps and hashsets are through Wikipedia dumps. I did some analysis of the most linked-to pages. Countries were among the highest. Hash sets for holding outgoing edges in the link graph ended up causing my program to exceed my laptop’s memory. Plain old lists (Python) were fine, though. And given there aren’t a crazy number of links per page using lists is fine performance wise.
- angrais 3y ago
  Why did lists require less memory? Was it because you only held a subset of keys in the lists?
  - teaearlgraycold 3y ago
    Lists in Python have a integer for their size and a pointer for each element. Sets presumably have some number of buckets that are used to put pointers in, but many more buckets are allocated than get used in small sets.
- wenyuanyu 3y ago
  Sort them, and use a vector of vectors for the adjacency list... Or better still use a graph processing library or graph database to manage that for you...
- marginalia_nu 3y ago
  This is a fairly large data set indeed. The memory overhead (which is probably something like 4-8x for hash maps?) can start to become fairly noticeable at those sizes. Since Wikipedia posts already have a canonical numeric ID, if map semantics are important, I'd probably load that mapping into memory and use something like roaringbitmap for compressed storage of relations.

dojitza1 3y ago

An interesting solution to the blind spot error (taken directly from Jeremy Howard's amazing guide to language models - https://www.youtube.com/watch?v=jkrNMKz9pWU https://www.youtube.com/watch?v=jkrNMKz9pWU) is to erase the chat history and try again. Once GPT has made an error (or as the author of this article says, the early layers have irreversibly pruned some important data), it will very often start to be even more wrong.

cypherpunks01 3y ago
When this happens, I'll usually say something along the lines of: "This isn't working and I'd like to start this again with a new ChatGPT conversation. Can you suggest a new improved prompt to complete this task, that takes into account everything we've learned so far?" It has given me good prompt suggestions that can immediately get a script working on the first try, after a frustrating series of blind spot bugs.
- jondwillis 3y ago
  I’ll give this a try when it undoubtedly happens to me later today while debugging something ;)
- sebastiennight 3y ago
  It seems surprising that this would work, because in my experience these LLMs don't really have good prompt-crafting skills. Can you please share a ChatGPT example where that was successful, including having the new prompt outperform the old one?
  - mercer 3y ago
    I've also not had much success with asking it to craft prompts.
- kridsdale3 3y ago
  I do a similar thing when the latest GPT+DALLE version says "I'm sorry I can't make a picture of that because it would violate content standards" (yesterday, this was because I asked for a visualization of medication acting to reduce arterial plaque. I can only assume arteries in the body ended up looking like dicks) So I say "Ok, let's start over. Rewrite my prompt in a way that minimizes the chance of the resulting image producing something that would trigger content standards checking"

lynx23 3y ago

For the author: Have you looked at the casing (Text.Casing) Hackage package?

gwern 3y ago
https://hackage.haskell.org/package/casing-0.1.4.1/docs/Text-Casing.html https://hackage.haskell.org/package/casing-0.1.4.1/docs/Text... ? This looks intended only for source code, and doesn't help with natural language text like page or paper titles.

k2enemy 3y ago

Pedantic, but gwern is looking for initialisms, not acronyms. Acronyms are pronounced as a word. https://www.merriam-webster.com/grammar/whats-an-acronym https://www.merriam-webster.com/grammar/whats-an-acronym

throw555chip 3y ago
> looking for initialisms, not acronyms Imprecise wording, initialisms are a case of acronyms, it's not either or. https://wwwnc.cdc.gov/eid/page/abbreviations-acronyms-initialisms https://wwwnc.cdc.gov/eid/page/abbreviations-acronyms-initia... "an initialism is an acronym that is pronounced as individual letters" https://www.writersdigest.com/write-better-fiction/abbreviation-vs-acronym-vs-initialism-grammar-rules https://www.writersdigest.com/write-better-fiction/abbreviat... "As such, acronyms are initialisms."
- gameman144 3y ago
  Wait, am I crazy or are these two articles saying the exact opposite thing about which class is the parent one? The CDC one seems to say that initialisms are a class of acronym, but the Writers Digest one says acronyms are a class of initialism.
  - Kerbonut 3y ago
    I read those the same way. In any case, both acronyms and initialisms are individual subsets of abbreviations.
    - jahnu 3y ago
      TIL
  - sebastiennight 3y ago
    The CDC link says they are two separate classes (one is pronounced as a word, the other one is pronounced by reading the letters) The Writer's Digest link says that initialisms are the parent class, and that acronyms are the special case of specifically pronouncing the letters as a word. So, root comment is correct (gwern is looking for initialisms) and GP is incorrect (initialisms are not a subset of acronyms in either definition linked by GP).

kuba-orlik 3y ago

Couldn't the same thing be achieved with grep?

gwern 3y ago
How? What are you going to grep? /usr/share/dict/words? Doesn't include all the proper nouns or organizations that WP would.

throw310822 3y ago

Funnily enough, there is a Wikipedia page with all three letter acronyms, that correctly shows CQK as the first unused one (red link). https://en.m.wikipedia.org/wiki/Wikipedia:TLAs_from_AAA_to_DZZ https://en.m.wikipedia.org/wiki/Wikipedia:TLAs_from_AAA_to_D...

gwern 3y ago
Hah! I didn't know that existed. Figuring out how to parse it would be a bit tricky, however... looking at the source, I think you could try to grep for 'title="CQK (page does not exist)"' and parse out the '[A-Z][A-Z][A-Z]? ' match to get the full list of absent TLAs and then negate for the present ones.