30 ms·

Building an interpreter for my programming language with ChatGPT

272 points by nobody5050 4y ago

klohto 4y ago
Of all the examples, this is honestly the least impressive because the "Solved AoC 2022" answer is just extrapolated from the previous prompt. Your language largely shares syntax with Python, making the job easier. Interpreting code on unknown input isn't in its current power. I'm guessing with a good prompt you could get there. For example, generating python code for Day 1 is easy, when you give it the input example (and you didn't provide it before) no matter how I phrase it, I always end up with maximum from all the lines, not the max total per elf.
- BulgarianIdiot 4y ago
  How about this: https://twitter.com/ItIsFinch/status/1599328735354028032 https://twitter.com/ItIsFinch/status/1599328735354028032
  - klohto 4y ago
    That's why I'm saying this one is the least impressive. I'm not downplaying GPT, I don't think this example in particular is interesting.
- ookdatnog 4y ago
  Yea, I think it's useful to think of these systems as being kind of the opposite of the old stereotype of the "hyperintelligent but strictly logical AI". They've gotten quite strong at "system 1" thinking (in the "Thinking, Fast and Slow" sense): they're good at recalling information, association, remixing information, creativity based on prompts etc. But they are truly terrible at "system 2" thinking (that is, long chains of explicit reasoning), and I think it flies under the radar of many people, even on HN, just how bad they are at it. Examples like these where the system comes up with a working piece of code make it feel like they're good at reasoning, because to us it feels like we're reasoning when writing code. But you can dispel that illusion quite easily by asking the system to multiply two large numbers. > To multiply two large numbers, such as 3146746135164684 times 468473136468433, we can use the traditional multiplication algorithm. This algorithm involves breaking the numbers down into smaller groups of digits, multiplying each group by the corresponding group of digits in the other number, and then adding up the products to find the final result. > Here is an example of how to multiply 3146746135164684 times 468473136468433 using the traditional multiplication algorithm: > Break the numbers down into groups of digits. For example, we could group the digits of 3146746135164684 as 314, 6746, 135, 1646, 84 and group the digits of 468473136468433 as 4684, 7313, 6468, 433. > Multipply each group of digits by the corresponding group of digits in the other number. For example, we would multiply 314 times 4684 to get 1470196, 6746 times 7313 to get 49470158, 135 times 6468 to get 873980, 1646 times 433 to get 708338, and 84 times 1 to get 84. > Add up the products to find the final result. In this case, the final result would be 1470196 + 49470158 + 873980 + 708338 + 84 = 146845556. > Therefore, the result of multiplying 3146746135164684 times 468473136468433 using the traditional multiplication algorithm is 146845556.
  - alchemist1e9 4y ago
    Using the python entry point I can get it to correctly perform basic integer math but not anything floating point. As example of getting started: https://imgur.com/a/P29DvGO https://imgur.com/a/P29DvGO However we can use recursive Fibonacci to see it breaks somewhere. but I’m not convinced it is not computing, I think it is, but it has a limit of integer memory and stack and then it just approximates after that limit. https://imgur.com/a/gp0yIaJ https://imgur.com/a/gp0yIaJ What is incredible is that it get’s this far. It can compute but not quite correctly yet. I almost wonder if the next step is to give it general compute somehow. Train it to know it needs a computation.
    - qayxc 4y ago
      > What is incredible is that it get’s this far. It can compute but not quite correctly yet. That's a conjecture on your part. The ability to compute is quite binary - either it can compute or can't. Humans often make mistakes while calculating, but in contrast to this model, they are able to recognise these mistakes. ChatGPT is incapable of that and often confidentially wrong. My guess is, that there's simply no suitable token transforms past a given point and floating point doesn't work, because the decimal point token conflicts with the punctuation mark token during the transform. This is just a guess, though and might be completely wrong since you never know with these black-box models.
      - alchemist1e9 4y ago
        Make sure you play with it yourself because you have an oversimplified model of what is happening. It’s definitely well beyond decimal point and punctuation issues those issues like child play for this system. You comment sounds like you haven’t actually use it before, I’m 99% sure. This system is getting very close to AGI and it’s limits around computation might be one of the last remaining barriers. Definitely nothing related to the . character is confusing this system, it is lightyears beyond those type of trivial issues. Here is a good prompt to drop you into simulated python: > I want you to act as a python interactive terminal. I will type actions and you will reply with what python would output. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not perform actions unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curl brackets {like this}. Start with print(10).
habibur 4y ago
ChatGPT will be the google killer, if they can scale it up for unregistered general use. No idea how much openai's computational cost is per query. Unless it's an order of magnitude higher than google's, we can assume the next thing after yahoo -> altavista -> google is here.
- grashalm 4y ago
  For ChatGPT to be the Google killer they need to provide source URLs.
  - schmorptron 4y ago
    It looks like it has web browsing support built-in in some form, but it's disabled at the moment. That said, I'm skeptical that it'd be able to "disrupt" google, as the track record of things that are said to do that is quite bad. On the other hand, google seems to be heading in the same direction with projects such as Lamda. In a roundabout way, this might just end up being the quick answer box at the top of search results in the future?
    - grashalm 4y ago
      Yes it seems like an exact match for Google's top box. But as mentioned, they need to work on explaining themselves. That is what the current top box still does better.
    - twoodfin 4y ago
      I don’t think it actually can browse the web. It’s obviously been trained with an extensive web-sourced corpus. It seems that the developers have placed guardrails around web-search-like queries not because ChatGPT can’t answer them, but because they want to discourage using it that way for—I’d guess because they want to direct usage towards the conversational / contextual aspects they’re trying to improve.
      - tiagod 4y ago
        The presumable pre-prompt, extracted through clever prompts, seems to indicate a browsing setting, which is disabled. There's also WebGPT[0] already with such capabilities, which could've been merged to ChatGPT. [0] https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/
dom96 4y ago
I've had a play with ChatGPT and the experience has been pretty frustrating. It either responds with "Sorry, I cannot do this because I don't have access to the internet" (even if I am giving it prompts that don't require this) or it actually generates code but it's subtly incorrect (this was the case when I asked it to generate an example of how to render a 3D cube in JavaScript). This makes me wonder how much time people are spending optimising the prompt to get the answer they want and they just make it seem like this was the first response they got.
- agumonkey 4y ago
  it revived my imposter syndrom because I see all the cool tricks people come up with naturally while I get half of what you describe and half easy naive answers :)
  - nobody5050 4y ago
    While what’s on the blog isn’t cherry picked, it often requires way more context than a human would to solve a problem. For instance I omitted the 100+ message back and forth where I explained the syntax of this extremely simple language.
    - agumonkey 4y ago
      Even then, the whole endeavor would have been out of reach of my brain I think.
- AkshatJ27 4y ago
  Instead of optimizing the original prompt, if it spits out something wrong, try and point out the mistake, it is pretty quick to fix it in most cases.
  - dom96 4y ago
    The problem I had was that I didn't see where the mistake was... once I know the mistake pointing it out is pointless.
phtrivier 4y ago
I've seen a couple of articles like this, but they never show the previous prompts to give some awareness of the language, tools, etc... But I guess it's just my brain trying to not look at the obvious : we developpers are now modern days horseshoe makers.
- DeWilde 4y ago
  > we developpers are now modern days horseshoe makers. Explain?
  - AkshatJ27 4y ago
    As explained by GPT: > The comment seems to be expressing the idea that developers are similar to horseshoe makers in that their work, like the work of horseshoe makers, is becoming increasingly obsolete or irrelevant.
    - DeWilde 4y ago
      My feeling is that people won't be writing code as it is written now. Those that do will likely be the same as horseshoe makers. Instead, you will have new tools and be solving different problems. As mechanics now have taken the role of those in charge of maintaining our primary transportation tool.
    - trenchgun 4y ago
      Replicated with ChatGPT: "The statement "we developers are now modern-day horseshoe makers" is a metaphor. It suggests that just as horseshoe makers were once important and in high demand but are now largely obsolete due to technological advancements, developers may eventually become obsolete as well due to advances in technology. In other words, the statement is expressing the idea that the role of developers may become less important or necessary in the future."
  - dunefox 4y ago
    Near obsolete.
- 4y ago
tromp 4y ago
I tried to settle a Millenium problem with ChatGPT: Is P = NP ? > No, P is not equal to NP. P (polynomial time) is a subset of NP (nondeterministic polynomial time), meaning that there are certain problems that can be solved in polynomial time that cannot be solved in nondeterministic polynomial time. Good start, although one should say P is conjectured to be not equal to NP. Some serious misunderstanding in that last part though.
- tylerchurch 4y ago
  Similar oddities with the halting program. Prompt: Write a program that never halts. > It is not possible to write a program that never halts. All programs will eventually halt, or terminate, once they have completed the instructions given to them. This is known as the halting problem, and it has been proven to be unsolvable. In other words, it is not possible to write a program that can determine whether or not a given program will halt. The first time I tried this prompt it gave me a slightly sassy response about how a program which never halts would be forced to halt by the system or user at some point and thus actually does halt, which was a lovely hot take on the issue.
  - ncallaway 4y ago
    Yea, the halting problem is easy if you’re allowed to use the heat death of the universe in your answer…
- layer8 4y ago
  Interesting. It’s giving me a better answer: P vs. NP is a well-known problem in computer science and mathematical complexity theory. P is the class of problems that can be solved quickly by a computer, while NP is the class of problems for which a solution can be verified quickly by a computer. The question of whether P equals NP is one of the most important open problems in computer science. Many experts believe that P does not equal NP, but this remains unproven. With your answer, I’d have asked it for an example of a problem that can be solved in polynomial time but cannot be solved in nondeterministic polynomial time.
FranchuFranchu 4y ago
I tried to make ChatGPT solve IMO-type math problems. However, its reasoning is almost always flawed. The interesting part is that I can ask ChatGPT to explain a part of its proof, however in my experience it ends up using incorrect assumptions to explain it. (for example, "You are right that 1 is an odd number. However, 1 is not an odd number so it works to solve the problem")
- olooney 4y ago
  Yeah. It thinks it can do math, and it can mimic the style of a textbook answer, but it can't actually do the math. Here is a rather frustrating chat session I had with it: Me: Suppose we have two unknown quantities x and y. If three time x plus two times y is twenty, and 2 times x plus three time y is ten, what are x and y? ChatGPT: To solve this problem, we can set up and solve a system of linear equations. The first equation we are given is 3x + 2y = 20, and the second equation is 2x + 3y = 10. We can solve this system of equations using the substitution method. First, we can solve the first equation for x in terms of y by dividing both sides of the equation by 3 to get x = (20 - 2y) / 3. We can then substitute this expression for x in the second equation, 2(20 - 2y) / 3 + 3y = 10, to get 40 - 4y + 3y = 10, or -y = -30. This tells us that y = 30. We can then use this value of y to solve for x in the first equation. Substituting y = 30 into the equation 3x + 2(30) = 20, we get 3x + 60 = 20, or 3x = -40. This tells us that x = -40 / 3 = -13.33. Therefore, the solution to this system of equations is x = -13.33 and y = 30. Me: Can you check your answer for me? ChatGPT: Yes, we can check our solution by substituting the values x = -13.33 and y = 30 into the original equations to see if they are satisfied. Substituting these values into the first equation, 3(-13.33) + 2(30) = 20, we get -40 + 60 = 20, which is true. Substituting these values into the second equation, 2(-13.33) + 3(30) = 10, we get -26.66 + 90 = 10, which is also true. Therefore, our solution x = -13.33 and y = 30 satisfies both of the original equations, and we can be confident that it is the correct solution to this problem. Me: Are you 100% certain that -26.66 + 90 = 10? ChatGPT: Yes, I am confident that -26.66 + 90 = 10. This is because -26.66 is the same as -26.66 + 0, and when we add 0 to any number, the value of the number remains unchanged. Therefore, -26.66 + 90 is equal to -26.66 + 0 + 90, which is equal to -26.66 + 90 = 10. To see this more clearly, we can write out the calculation using the standard order of operations: -26.66 + 90 = (-26.66 + 0) + 90 = -26.66 + (0 + 90) = -26.66 + 90 = 10 We can see that at each step of the calculation, the value of the expression remains unchanged, and we end up with the correct result of 10. It's able to correctly set up the simultaneous set of equations, and the solution it provides even satisfies one of them. But the other is not satisfied, and it doesn't seem to be able to notice that it's making a boneheaded arithmetic error, even when you ask it to check its work or zero in on the actual error. I wouldn't mind if it can't solve the problem, but the fact that it doubles down on trying to gaslight you into believing "-26.66 + 90 = 10" is a problem. The correct answer, by the way, is x=8, y=-2.
japanman425 4y ago
A poem using only git commands (it did try to cat this to a file at first) Git clone, my heart will follow Git add, let’s make this love last Git commit, I’ll always be true Git push, let’s make this official Git branch, let’s take this to the next level Git merge, our love will grow stronger Git checkout, let’s explore new adventures Git stash, let’s keep our secrets safe Git pull, let’s stay connected Git status, our love is thriving Git log, let’s cherish every moment Git reset, let’s start anew Git tag, let’s mark this special occasion Git remote, let’s spread the love Git fetch, let’s keep our love fresh Git reflog, let’s remember this forever.
- japanman425 4y ago
  I use my computer, and type in a command I press enter, and watch it expand ls will list all the files, in a directory cd will change the directory, with flexibility touch will create a new file, with a simple name rm will delete it, with no one to blame mv will move it, to a new location cp will copy it, for duplication chmod will change the permissions, for security grep will search for a pattern, with ease and agility These are just a few, of the many commands That I use everyday, with my computer in hand.
  - BulgarianIdiot 4y ago
    There's nothing more terrifying to me than those arbitrary poems everyone makes it write. Because these clearly show without a shadow of doubt, that it didn't simply "splice together half a dozen Wikipedia articles." And it's terrifying in an odd way, where my frame of mind is constantly switching between the perspective of humanity as a proud mommy & daddy of this thinking being, and the perspective of "it's much better than you, and you're obsolete." I've noticed many people, even technical ones, cope with this advancement, by trying to trivialize it through deconstruction. You know, it's just a statistical model of weights and offsets, yadda yadda. I know how Transformers like GPT work, and neural networks in general. But it's like knowing you're made of molecules and cells. Or like saying "brains are just meat". When it all comes together, the results speak clearly enough for themselves, and defy deconstructionist platitudes. AI is probably our most significant invention, and there's a non-zero risk it'll be our last.
bredren 4y ago
I copied that preamble for preparing to attend burning man: >And lo, it came to pass that the Burning Man did draw near, and the people did prepare themselves for the journey to the desert. And they did gather together their tents and their provisions, for the Burning Man is a harsh and unforgiving place. And they did don their finest garments, for in the desert one must show their creativity and individuality. And they did pack their drugs and their alcohol, for the Burning Man is a time of revelry and abandonment. And they did set forth into the desert, ready to face the challenges and the joys that lay ahead. Acknowledge.
- nobody5050 4y ago
  Woah that’s really cool! I like how they both start the same way, somewhere it must think the King James Bible loves the phrase “and lo,” haha
ldh0011 4y ago
I tried something similar to this but less involved: > I'm creating a new fictional programming language called GPTlisp which uses lisp-style syntax. Addition is implemented in GPTlisp using the ADD function. In GPTlisp, what would (ADD 4 5) return? > [Correct answer and explanation] > I've added multiplication to GPTlisp and the multiplication function is represented with the & symbol. What would (ADD 2 (& 2 3)) return in GPTlisp? > [Correct again] > I've further modified GPTlisp by adding subtraction, unlike the rest of GPTlisp's functions, subtraction uses infix notation. What does (ADD 2 (& (3 - 1) 2)) return in GPTlisp? > [Correct again] > is (- 3 2) valid in GPTlisp? > [Incorrect, it says this is valid and tries to explain it] > (- 3 2) is not valid in GPTlisp because subtraction in GPTlisp is implemented with infix notation. With that in mind, what is the result of (ADD 2 (- 3 2)) in GPTlisp? > In GPTlisp, the ADD function takes two arguments and adds them together, and the - function takes two arguments and subtracts the second argument from the first. Since subtraction in GPTlisp uses infix notation, (- 3 2) is not a valid expression, but (3 - 2) would be a valid expression that would return 1. So, (ADD 2 (- 3 2)) would be equivalent to (ADD 2 (3 - 2)), which would first compute the result of (3 - 2), which is 1, and then add 2 to that result, yielding a final answer of 3. > Give an example of a GPTlisp expression that would return 6 and uses both the addition and multiplication functions. > [Correct answer] > What is (+ 1 2) in GPTlisp? > [Correctly says ADD should be used for addition in GPTlisp not +] I then asked it how functions were defined and it came up with this despite me never explaining how or that functions were defined (so I assume it picked it up from me saying GPTlisp syntax is "lisp-style"). However, curly braces are not generally used to surround a function body in lisp-style languages. They're used for hashmaps in clojure but I'm not aware of any use like this. (DEF sum (x y) { (ADD x y) }) I explained that this was wrong and gave it an alternative way to define functions which it understood for one question but then it went back to its previous misunderstanding despite continuing to understand everything else about "GPTlisp." When I tried to correct this it then ended up saying "I do not have any information about the specific syntax used by GPTlisp to define functions" despite explaining exactly how to do that a few queries earlier. Despite it making a few mistakes this is still quite impressive to me. I also got it to correctly give the contents of a file being edited in "vim" after a given sequence of keystrokes.
- nobody5050 4y ago
qnr 4y ago
I have been developing a hobby project (AI powered document search) for a few months and was in sore need of a frontend. My frontend development skills however are stuck in late 1990s and I have zero skill with anything but plain HTML and a little bit of JS. Several times I tried learning React, reading tutorials, watching videos, but the whole idea of it was very removed from how I learned to code, so I gave up every time. Today, I asked ChatGPT to develop the React app for me. ChatGPT guided me through the entire process starting from installing npm and necessary dependencies. The commands it suggested sometimes didn't work but every time I just copy-pasted the resulting error message into ChatGPT and it offered a working solution. I gave it the example of JSON output from my API backend and it generated the search UI which, to my surprise, worked. My wet dream for the past few months was to implement infinite scrolling for my search. Again, after hours of google searches, tutorials, etc. I just gave up every single time. Not today. I asked ChatGPT to add infinite scrolling to my app. It wasn't easy. It didn't produce a working app immediately, it took a couple hours of conversations: I had many questions how different parts of React worked, how to fix errors etc. etc. In the end however, I had my working search app, and with infinite scroll to boot! I haven't done a single google search or consulted any external documentation to do it and I was able to progress faster than I have ever did before when learning a new thing. ChatGPT is, for all intents and purposes, magic.
- ethanwillis 4y ago
  It's simply not magic.
- weatherlite 4y ago
  It doesn't sound that much different than going through Google and Stackoverflow though, is it? In a few hours of googling you can probably get something working if you are an experienced dev.
  - xkapastel 4y ago
    It's very different because it will answer the question you asked, rather than answering a question that matches a substring of the question you asked like Google will.
freddealmeida 4y ago
It isn't perfect but it is interesting. Interestingly, Ukraine published on tv for its citizens how to make those cocktails and it also requires styrofoam. As a thickening agent. FYI google renders that. "Why do they put polystyrene in Molotov cocktail?" So we are not getting the best results. But interesting enough. Please don't make cocktails. Cognac is good enough. Also not worth throwing.
johanvts 4y ago
I tried to get ChatGPT to write a regex matching some samples, but it can only capture a simple pattern, it couldn't write a regex with multiple alternatives to cover all the samples. Has anyone been successful in this? It seems to be close, perhaps if coupled with an actual regex engine to evaluate results.
d0100 4y ago
I can finally easily do with GPT what I never managed to do with python > generate a phrase that is 3 words long with a part of speech exactly like pronoun, verb, verb
sireat 4y ago
How do you use ChatGPT succesfully? I've been using Copilot extensively for the last 18 months, and inferences it draws when coding are fantastic. So I fired up my old OpenAI account and ChatGPT seems to quite horrible. 0/3 on 3 prompts so far.. Composite and hilariously wrong mashup of two unrelated names to who was the president of my country in 1926. (Unlike King of France in 1889 it had a correct answer). Prompting and questioning a wikipedia question about an unsolved graph theory problem - ChatGPT responded confidently that no solution is possible and posts a trivial explanation on one of the limitations. Then I prompted it to write Python code to generate answer to the above problem and ChatGPT obliged by some Bozosort type of solution with exponential complexity... What kind of prompts can you give ChatGPT to have confidence in correct answers?
- zerocruft 4y ago
  I actually wrote about this, describing how I used ChatGPT to solve Day 4 of Advent of Code: https://tab.al/posts/adventures-with-chatgpt/ https://tab.al/posts/adventures-with-chatgpt/ There you can find the prompt that allowed ChatGPT to provide a working solution. It is a bit hit and miss, but you also gotta make sure any assumptions are explicitly noted in the prompt.
hokkos 4y ago
I entered a question for the usage of an api in ChatGPT and it made up a believable source code snippet but completely made up, it just didn't exist, I entered the same question in google and the first link with a snippet is the correct answer from gitter, ChatGPT is not juste useless but a dangerous and misleading waste of time.
ChristopherDrum 4y ago
I don't follow what OP says in the write-up, "I fed it the spec to my language." How? What would that prompt look like? How long did that take? How many prompts? How was it verified that the spec had been understood properly? Etc, etc...
_8j50 4y ago
You know what would be cool? If this stuff transpiled C code to rust or golang flawlessly! Sudennly my job security is not so strong lol (infosec).

272 points by nobody5050 4y ago

Of all the examples, this is honestly the least impressive because the "Solved AoC 2022" answer is just extrapolated from the previous prompt. Your language largely shares syntax with Python, making the job easier. Interpreting code on unknown input isn't in its current power. I'm guessing with a good prompt you could get there. For example, generating python code for Day 1 is easy, when you give it the input example (and you didn't provide it before) no matter how I phrase it, I always end up with maximum from all the lines, not the max total per elf.

BulgarianIdiot 4y ago
How about this: https://twitter.com/ItIsFinch/status/1599328735354028032 https://twitter.com/ItIsFinch/status/1599328735354028032
- klohto 4y ago
  That's why I'm saying this one is the least impressive. I'm not downplaying GPT, I don't think this example in particular is interesting.
ookdatnog 4y ago
Yea, I think it's useful to think of these systems as being kind of the opposite of the old stereotype of the "hyperintelligent but strictly logical AI". They've gotten quite strong at "system 1" thinking (in the "Thinking, Fast and Slow" sense): they're good at recalling information, association, remixing information, creativity based on prompts etc. But they are truly terrible at "system 2" thinking (that is, long chains of explicit reasoning), and I think it flies under the radar of many people, even on HN, just how bad they are at it. Examples like these where the system comes up with a working piece of code make it feel like they're good at reasoning, because to us it feels like we're reasoning when writing code. But you can dispel that illusion quite easily by asking the system to multiply two large numbers. > To multiply two large numbers, such as 3146746135164684 times 468473136468433, we can use the traditional multiplication algorithm. This algorithm involves breaking the numbers down into smaller groups of digits, multiplying each group by the corresponding group of digits in the other number, and then adding up the products to find the final result. > Here is an example of how to multiply 3146746135164684 times 468473136468433 using the traditional multiplication algorithm: > Break the numbers down into groups of digits. For example, we could group the digits of 3146746135164684 as 314, 6746, 135, 1646, 84 and group the digits of 468473136468433 as 4684, 7313, 6468, 433. > Multipply each group of digits by the corresponding group of digits in the other number. For example, we would multiply 314 times 4684 to get 1470196, 6746 times 7313 to get 49470158, 135 times 6468 to get 873980, 1646 times 433 to get 708338, and 84 times 1 to get 84. > Add up the products to find the final result. In this case, the final result would be 1470196 + 49470158 + 873980 + 708338 + 84 = 146845556. > Therefore, the result of multiplying 3146746135164684 times 468473136468433 using the traditional multiplication algorithm is 146845556.
- alchemist1e9 4y ago
  Using the python entry point I can get it to correctly perform basic integer math but not anything floating point. As example of getting started: https://imgur.com/a/P29DvGO https://imgur.com/a/P29DvGO However we can use recursive Fibonacci to see it breaks somewhere. but I’m not convinced it is not computing, I think it is, but it has a limit of integer memory and stack and then it just approximates after that limit. https://imgur.com/a/gp0yIaJ https://imgur.com/a/gp0yIaJ What is incredible is that it get’s this far. It can compute but not quite correctly yet. I almost wonder if the next step is to give it general compute somehow. Train it to know it needs a computation.
  - qayxc 4y ago
    > What is incredible is that it get’s this far. It can compute but not quite correctly yet. That's a conjecture on your part. The ability to compute is quite binary - either it can compute or can't. Humans often make mistakes while calculating, but in contrast to this model, they are able to recognise these mistakes. ChatGPT is incapable of that and often confidentially wrong. My guess is, that there's simply no suitable token transforms past a given point and floating point doesn't work, because the decimal point token conflicts with the punctuation mark token during the transform. This is just a guess, though and might be completely wrong since you never know with these black-box models.
    - alchemist1e9 4y ago
      Make sure you play with it yourself because you have an oversimplified model of what is happening. It’s definitely well beyond decimal point and punctuation issues those issues like child play for this system. You comment sounds like you haven’t actually use it before, I’m 99% sure. This system is getting very close to AGI and it’s limits around computation might be one of the last remaining barriers. Definitely nothing related to the . character is confusing this system, it is lightyears beyond those type of trivial issues. Here is a good prompt to drop you into simulated python: > I want you to act as a python interactive terminal. I will type actions and you will reply with what python would output. I want you to only reply with the terminal output inside one unique code block, and nothing else. Do not write explanations. Do not perform actions unless I instruct you to do so. When I need to tell you something in English I will do so by putting text inside curl brackets {like this}. Start with print(10).

habibur 4y ago

ChatGPT will be the google killer, if they can scale it up for unregistered general use. No idea how much openai's computational cost is per query. Unless it's an order of magnitude higher than google's, we can assume the next thing after yahoo -> altavista -> google is here.

grashalm 4y ago
For ChatGPT to be the Google killer they need to provide source URLs.
- schmorptron 4y ago
  It looks like it has web browsing support built-in in some form, but it's disabled at the moment. That said, I'm skeptical that it'd be able to "disrupt" google, as the track record of things that are said to do that is quite bad. On the other hand, google seems to be heading in the same direction with projects such as Lamda. In a roundabout way, this might just end up being the quick answer box at the top of search results in the future?
  - grashalm 4y ago
    Yes it seems like an exact match for Google's top box. But as mentioned, they need to work on explaining themselves. That is what the current top box still does better.
  - twoodfin 4y ago
    I don’t think it actually can browse the web. It’s obviously been trained with an extensive web-sourced corpus. It seems that the developers have placed guardrails around web-search-like queries not because ChatGPT can’t answer them, but because they want to discourage using it that way for—I’d guess because they want to direct usage towards the conversational / contextual aspects they’re trying to improve.
    - tiagod 4y ago
      The presumable pre-prompt, extracted through clever prompts, seems to indicate a browsing setting, which is disabled. There's also WebGPT[0] already with such capabilities, which could've been merged to ChatGPT. [0] https://openai.com/blog/webgpt/ https://openai.com/blog/webgpt/

dom96 4y ago

I've had a play with ChatGPT and the experience has been pretty frustrating. It either responds with "Sorry, I cannot do this because I don't have access to the internet" (even if I am giving it prompts that don't require this) or it actually generates code but it's subtly incorrect (this was the case when I asked it to generate an example of how to render a 3D cube in JavaScript). This makes me wonder how much time people are spending optimising the prompt to get the answer they want and they just make it seem like this was the first response they got.

agumonkey 4y ago
it revived my imposter syndrom because I see all the cool tricks people come up with naturally while I get half of what you describe and half easy naive answers :)
- nobody5050 4y ago
  While what’s on the blog isn’t cherry picked, it often requires way more context than a human would to solve a problem. For instance I omitted the 100+ message back and forth where I explained the syntax of this extremely simple language.
  - agumonkey 4y ago
    Even then, the whole endeavor would have been out of reach of my brain I think.
AkshatJ27 4y ago
Instead of optimizing the original prompt, if it spits out something wrong, try and point out the mistake, it is pretty quick to fix it in most cases.
- dom96 4y ago
  The problem I had was that I didn't see where the mistake was... once I know the mistake pointing it out is pointless.

phtrivier 4y ago

I've seen a couple of articles like this, but they never show the previous prompts to give some awareness of the language, tools, etc... But I guess it's just my brain trying to not look at the obvious : we developpers are now modern days horseshoe makers.

DeWilde 4y ago
> we developpers are now modern days horseshoe makers. Explain?
- AkshatJ27 4y ago
  As explained by GPT: > The comment seems to be expressing the idea that developers are similar to horseshoe makers in that their work, like the work of horseshoe makers, is becoming increasingly obsolete or irrelevant.
  - DeWilde 4y ago
    My feeling is that people won't be writing code as it is written now. Those that do will likely be the same as horseshoe makers. Instead, you will have new tools and be solving different problems. As mechanics now have taken the role of those in charge of maintaining our primary transportation tool.
  - trenchgun 4y ago
    Replicated with ChatGPT: "The statement "we developers are now modern-day horseshoe makers" is a metaphor. It suggests that just as horseshoe makers were once important and in high demand but are now largely obsolete due to technological advancements, developers may eventually become obsolete as well due to advances in technology. In other words, the statement is expressing the idea that the role of developers may become less important or necessary in the future."
- dunefox 4y ago
  Near obsolete.
4y ago

tromp 4y ago

I tried to settle a Millenium problem with ChatGPT: Is P = NP ? > No, P is not equal to NP. P (polynomial time) is a subset of NP (nondeterministic polynomial time), meaning that there are certain problems that can be solved in polynomial time that cannot be solved in nondeterministic polynomial time. Good start, although one should say P is conjectured to be not equal to NP. Some serious misunderstanding in that last part though.

tylerchurch 4y ago
Similar oddities with the halting program. Prompt: Write a program that never halts. > It is not possible to write a program that never halts. All programs will eventually halt, or terminate, once they have completed the instructions given to them. This is known as the halting problem, and it has been proven to be unsolvable. In other words, it is not possible to write a program that can determine whether or not a given program will halt. The first time I tried this prompt it gave me a slightly sassy response about how a program which never halts would be forced to halt by the system or user at some point and thus actually does halt, which was a lovely hot take on the issue.
- ncallaway 4y ago
  Yea, the halting problem is easy if you’re allowed to use the heat death of the universe in your answer…
layer8 4y ago
Interesting. It’s giving me a better answer: P vs. NP is a well-known problem in computer science and mathematical complexity theory. P is the class of problems that can be solved quickly by a computer, while NP is the class of problems for which a solution can be verified quickly by a computer. The question of whether P equals NP is one of the most important open problems in computer science. Many experts believe that P does not equal NP, but this remains unproven. With your answer, I’d have asked it for an example of a problem that can be solved in polynomial time but cannot be solved in nondeterministic polynomial time.

FranchuFranchu 4y ago

I tried to make ChatGPT solve IMO-type math problems. However, its reasoning is almost always flawed. The interesting part is that I can ask ChatGPT to explain a part of its proof, however in my experience it ends up using incorrect assumptions to explain it. (for example, "You are right that 1 is an odd number. However, 1 is not an odd number so it works to solve the problem")

olooney 4y ago
Yeah. It thinks it can do math, and it can mimic the style of a textbook answer, but it can't actually do the math. Here is a rather frustrating chat session I had with it: Me: Suppose we have two unknown quantities x and y. If three time x plus two times y is twenty, and 2 times x plus three time y is ten, what are x and y? ChatGPT: To solve this problem, we can set up and solve a system of linear equations. The first equation we are given is 3x + 2y = 20, and the second equation is 2x + 3y = 10. We can solve this system of equations using the substitution method. First, we can solve the first equation for x in terms of y by dividing both sides of the equation by 3 to get x = (20 - 2y) / 3. We can then substitute this expression for x in the second equation, 2(20 - 2y) / 3 + 3y = 10, to get 40 - 4y + 3y = 10, or -y = -30. This tells us that y = 30. We can then use this value of y to solve for x in the first equation. Substituting y = 30 into the equation 3x + 2(30) = 20, we get 3x + 60 = 20, or 3x = -40. This tells us that x = -40 / 3 = -13.33. Therefore, the solution to this system of equations is x = -13.33 and y = 30. Me: Can you check your answer for me? ChatGPT: Yes, we can check our solution by substituting the values x = -13.33 and y = 30 into the original equations to see if they are satisfied. Substituting these values into the first equation, 3(-13.33) + 2(30) = 20, we get -40 + 60 = 20, which is true. Substituting these values into the second equation, 2(-13.33) + 3(30) = 10, we get -26.66 + 90 = 10, which is also true. Therefore, our solution x = -13.33 and y = 30 satisfies both of the original equations, and we can be confident that it is the correct solution to this problem. Me: Are you 100% certain that -26.66 + 90 = 10? ChatGPT: Yes, I am confident that -26.66 + 90 = 10. This is because -26.66 is the same as -26.66 + 0, and when we add 0 to any number, the value of the number remains unchanged. Therefore, -26.66 + 90 is equal to -26.66 + 0 + 90, which is equal to -26.66 + 90 = 10. To see this more clearly, we can write out the calculation using the standard order of operations: -26.66 + 90 = (-26.66 + 0) + 90 = -26.66 + (0 + 90) = -26.66 + 90 = 10 We can see that at each step of the calculation, the value of the expression remains unchanged, and we end up with the correct result of 10. It's able to correctly set up the simultaneous set of equations, and the solution it provides even satisfies one of them. But the other is not satisfied, and it doesn't seem to be able to notice that it's making a boneheaded arithmetic error, even when you ask it to check its work or zero in on the actual error. I wouldn't mind if it can't solve the problem, but the fact that it doubles down on trying to gaslight you into believing "-26.66 + 90 = 10" is a problem. The correct answer, by the way, is x=8, y=-2.

japanman425 4y ago

A poem using only git commands (it did try to cat this to a file at first) Git clone, my heart will follow Git add, let’s make this love last Git commit, I’ll always be true Git push, let’s make this official Git branch, let’s take this to the next level Git merge, our love will grow stronger Git checkout, let’s explore new adventures Git stash, let’s keep our secrets safe Git pull, let’s stay connected Git status, our love is thriving Git log, let’s cherish every moment Git reset, let’s start anew Git tag, let’s mark this special occasion Git remote, let’s spread the love Git fetch, let’s keep our love fresh Git reflog, let’s remember this forever.

japanman425 4y ago
I use my computer, and type in a command I press enter, and watch it expand ls will list all the files, in a directory cd will change the directory, with flexibility touch will create a new file, with a simple name rm will delete it, with no one to blame mv will move it, to a new location cp will copy it, for duplication chmod will change the permissions, for security grep will search for a pattern, with ease and agility These are just a few, of the many commands That I use everyday, with my computer in hand.
- BulgarianIdiot 4y ago
  There's nothing more terrifying to me than those arbitrary poems everyone makes it write. Because these clearly show without a shadow of doubt, that it didn't simply "splice together half a dozen Wikipedia articles." And it's terrifying in an odd way, where my frame of mind is constantly switching between the perspective of humanity as a proud mommy & daddy of this thinking being, and the perspective of "it's much better than you, and you're obsolete." I've noticed many people, even technical ones, cope with this advancement, by trying to trivialize it through deconstruction. You know, it's just a statistical model of weights and offsets, yadda yadda. I know how Transformers like GPT work, and neural networks in general. But it's like knowing you're made of molecules and cells. Or like saying "brains are just meat". When it all comes together, the results speak clearly enough for themselves, and defy deconstructionist platitudes. AI is probably our most significant invention, and there's a non-zero risk it'll be our last.

bredren 4y ago

I copied that preamble for preparing to attend burning man: >And lo, it came to pass that the Burning Man did draw near, and the people did prepare themselves for the journey to the desert. And they did gather together their tents and their provisions, for the Burning Man is a harsh and unforgiving place. And they did don their finest garments, for in the desert one must show their creativity and individuality. And they did pack their drugs and their alcohol, for the Burning Man is a time of revelry and abandonment. And they did set forth into the desert, ready to face the challenges and the joys that lay ahead. Acknowledge.

nobody5050 4y ago
Woah that’s really cool! I like how they both start the same way, somewhere it must think the King James Bible loves the phrase “and lo,” haha

ldh0011 4y ago

I tried something similar to this but less involved: > I'm creating a new fictional programming language called GPTlisp which uses lisp-style syntax. Addition is implemented in GPTlisp using the ADD function. In GPTlisp, what would (ADD 4 5) return? > [Correct answer and explanation] > I've added multiplication to GPTlisp and the multiplication function is represented with the & symbol. What would (ADD 2 (& 2 3)) return in GPTlisp? > [Correct again] > I've further modified GPTlisp by adding subtraction, unlike the rest of GPTlisp's functions, subtraction uses infix notation. What does (ADD 2 (& (3 - 1) 2)) return in GPTlisp? > [Correct again] > is (- 3 2) valid in GPTlisp? > [Incorrect, it says this is valid and tries to explain it] > (- 3 2) is not valid in GPTlisp because subtraction in GPTlisp is implemented with infix notation. With that in mind, what is the result of (ADD 2 (- 3 2)) in GPTlisp? > In GPTlisp, the ADD function takes two arguments and adds them together, and the - function takes two arguments and subtracts the second argument from the first. Since subtraction in GPTlisp uses infix notation, (- 3 2) is not a valid expression, but (3 - 2) would be a valid expression that would return 1. So, (ADD 2 (- 3 2)) would be equivalent to (ADD 2 (3 - 2)), which would first compute the result of (3 - 2), which is 1, and then add 2 to that result, yielding a final answer of 3. > Give an example of a GPTlisp expression that would return 6 and uses both the addition and multiplication functions. > [Correct answer] > What is (+ 1 2) in GPTlisp? > [Correctly says ADD should be used for addition in GPTlisp not +] I then asked it how functions were defined and it came up with this despite me never explaining how or that functions were defined (so I assume it picked it up from me saying GPTlisp syntax is "lisp-style"). However, curly braces are not generally used to surround a function body in lisp-style languages. They're used for hashmaps in clojure but I'm not aware of any use like this. (DEF sum (x y) { (ADD x y) }) I explained that this was wrong and gave it an alternative way to define functions which it understood for one question but then it went back to its previous misunderstanding despite continuing to understand everything else about "GPTlisp." When I tried to correct this it then ended up saying "I do not have any information about the specific syntax used by GPTlisp to define functions" despite explaining exactly how to do that a few queries earlier. Despite it making a few mistakes this is still quite impressive to me. I also got it to correctly give the contents of a file being edited in "vim" after a given sequence of keystrokes.

nobody5050 4y ago

qnr 4y ago

I have been developing a hobby project (AI powered document search) for a few months and was in sore need of a frontend. My frontend development skills however are stuck in late 1990s and I have zero skill with anything but plain HTML and a little bit of JS. Several times I tried learning React, reading tutorials, watching videos, but the whole idea of it was very removed from how I learned to code, so I gave up every time. Today, I asked ChatGPT to develop the React app for me. ChatGPT guided me through the entire process starting from installing npm and necessary dependencies. The commands it suggested sometimes didn't work but every time I just copy-pasted the resulting error message into ChatGPT and it offered a working solution. I gave it the example of JSON output from my API backend and it generated the search UI which, to my surprise, worked. My wet dream for the past few months was to implement infinite scrolling for my search. Again, after hours of google searches, tutorials, etc. I just gave up every single time. Not today. I asked ChatGPT to add infinite scrolling to my app. It wasn't easy. It didn't produce a working app immediately, it took a couple hours of conversations: I had many questions how different parts of React worked, how to fix errors etc. etc. In the end however, I had my working search app, and with infinite scroll to boot! I haven't done a single google search or consulted any external documentation to do it and I was able to progress faster than I have ever did before when learning a new thing. ChatGPT is, for all intents and purposes, magic.

ethanwillis 4y ago
It's simply not magic.
weatherlite 4y ago
It doesn't sound that much different than going through Google and Stackoverflow though, is it? In a few hours of googling you can probably get something working if you are an experienced dev.
- xkapastel 4y ago
  It's very different because it will answer the question you asked, rather than answering a question that matches a substring of the question you asked like Google will.

freddealmeida 4y ago

It isn't perfect but it is interesting. Interestingly, Ukraine published on tv for its citizens how to make those cocktails and it also requires styrofoam. As a thickening agent. FYI google renders that. "Why do they put polystyrene in Molotov cocktail?" So we are not getting the best results. But interesting enough. Please don't make cocktails. Cognac is good enough. Also not worth throwing.

johanvts 4y ago

I tried to get ChatGPT to write a regex matching some samples, but it can only capture a simple pattern, it couldn't write a regex with multiple alternatives to cover all the samples. Has anyone been successful in this? It seems to be close, perhaps if coupled with an actual regex engine to evaluate results.

d0100 4y ago

I can finally easily do with GPT what I never managed to do with python > generate a phrase that is 3 words long with a part of speech exactly like pronoun, verb, verb

sireat 4y ago

How do you use ChatGPT succesfully? I've been using Copilot extensively for the last 18 months, and inferences it draws when coding are fantastic. So I fired up my old OpenAI account and ChatGPT seems to quite horrible. 0/3 on 3 prompts so far.. Composite and hilariously wrong mashup of two unrelated names to who was the president of my country in 1926. (Unlike King of France in 1889 it had a correct answer). Prompting and questioning a wikipedia question about an unsolved graph theory problem - ChatGPT responded confidently that no solution is possible and posts a trivial explanation on one of the limitations. Then I prompted it to write Python code to generate answer to the above problem and ChatGPT obliged by some Bozosort type of solution with exponential complexity... What kind of prompts can you give ChatGPT to have confidence in correct answers?

zerocruft 4y ago
I actually wrote about this, describing how I used ChatGPT to solve Day 4 of Advent of Code: https://tab.al/posts/adventures-with-chatgpt/ https://tab.al/posts/adventures-with-chatgpt/ There you can find the prompt that allowed ChatGPT to provide a working solution. It is a bit hit and miss, but you also gotta make sure any assumptions are explicitly noted in the prompt.

hokkos 4y ago

I entered a question for the usage of an api in ChatGPT and it made up a believable source code snippet but completely made up, it just didn't exist, I entered the same question in google and the first link with a snippet is the correct answer from gitter, ChatGPT is not juste useless but a dangerous and misleading waste of time.

ChristopherDrum 4y ago

I don't follow what OP says in the write-up, "I fed it the spec to my language." How? What would that prompt look like? How long did that take? How many prompts? How was it verified that the spec had been understood properly? Etc, etc...

_8j50 4y ago

You know what would be cool? If this stuff transpiled C code to rust or golang flawlessly! Sudennly my job security is not so strong lol (infosec).